Computer architecture and process of patient generation, evolution, and simulation for computer based testing system using bayesian networks as a scripting language

ABSTRACT

A method and system for patient generation and evolution for a computer-based testing system and/or expert system. One or more belief networks, which describe parallel health state networks are accessed by a user or a computer. A knowledge base, at least in part, is scripted from the one or more belief networks by the computer. A model patient at least in part, is instantiated by the computer from the scripted knowledge base. Optionally, the model patient is evolved by the computer in accordance with the parallel health state networks and responsive to a received course of action.

RELATED APPLICATIONS

This application is a continuation of U.S. Ser. No. 10/265,599, filedOct. 8, 2002, which is a divisional of U.S. Ser. No. 09/521,242, filedApr. 5, 2000, which claims priority to U.S. Ser. No. 60/127,850, filedApr. 5, 1999, the details of which are hereby incorporated by reference.

FIELD OF THE INVENTION

The present invention is generally related to a computer architectureand process for patient generation, evolution, and simulation, and moreparticularly to a computer architecture and process for patientgeneration, evolution, and simulation for a computer based testingsystem using belief networks and/or causal probabilistic networks as ascripting language.

BACKGROUND OF THE RELATED ART

Medical certifying organizations have traditionally relied upon paperand pencil cognitive examinations as a method for the assessment of thecandidate's medical knowledge. Traditional formats such as multiplechoice questions have well-defined operating characteristics andreliability for examining cognitive knowledge capabilities. See, forexample, Stocking M L, An alternative method for scoring adaptive tests,Research Report RR-94-98, 1994, incorporated herein by reference.

However, these tools generally measure in only cognitive knowledge.These methods provide only primitive ability to assess a candidate'sproblem-solving abilities. See, for example, Stillman P L, Swanson D B,Ensuring the clinical competence of medical school graduates throughstandardized patients, Arch Int Med 1978, Vol. 147, pages 1049-52,incorporated herein by reference.

Several organizations have previously experimented withcomputer-delivery of clinical content and evaluation. In the late 1960sand 1970s, the Ohio State University developed a self-directedIndependent Study Program which utilized a “Tutorial Evaluation System,”for conveying curriculum content. See, for example, Weinberg A D, CAI atthe Ohio State University College of Medicine, Comput Biol Med 1973,Vol. 3, pages 299-305; Merola A J, Pengov R E, Stokes B T,Computer-supported independent study in the basic medical sciences in:DeLand E C (ed). Information Technology in Health Science Education,Plenum Press, New York, 1973, incorporated herein by reference.

Co-synchronously Dr. Octo Barnett's laboratory at the MassachusettsGeneral hospital began development of clinical simulations. See, forexample, Barnett G O, The use of a computer-based system to teachclinical problem-solving, Computers in Biomedical Research, AcademicPress, New York 1974; Vol. 4, pages 301-19; Barnett G O, Hoffer E P,Famiglieti K T, Computers in medical education: present and future,Proceedings of the Seventh Annual Symposium on Computer Applications inMedical Care, IEEE Press, Washington, D.C. 1983, pages 11-13,incorporated herein by reference. The clinical simulations used theMUMPS language.

At approximately the same time, investigators at the University ofIllinois developed a simulation model known as (Computer-AssociatedSimulation of the Clinical Encounter, or “CASE”). See, for example,Harless W G, Farr N A, Zier M A, et al., MERIT—an application of CASE,Deland E C (ed), Information Technology in Health Science Education,Plenum Press, New York 1978, pages 565-69, incorporated herein byreference. This system was at one time considered by the American Boardof Internal Medicine (ABIM) as at least one component of arecertification process. Friedman R B, A computer program for simulatingthe patient-physician encounter, J Med Educ 1973, Vol. 48, pages 92-7,incorporated herein by reference. Research supported by the ABIMdemonstrated that a computerized examination system appeared feasible inprofessional evaluation/certification settings. Reshetar, R A, et al.,An Adaptive Testing Simulation for a Certifying Examination, presentedat the Annual Meeting of the American Educational Research Association,San Francisco, Calif., April, 1992, incorporated herein by reference.

Stevens and colleagues have also demonstrated the feasibility of usingcomputer-based systems for testing problem-solving ability inundergraduate medical school curriculum applications. See, for example,Stevens R H, et al, Evaluating Preclinical Medical Students by UsingComputer-Based Problem-Solving Examinations, Academic Medicine 1989,Volume 64, pages 685-87, incorporated herein by reference. Sittig andcolleagues have also examined the utility of computer-based instructionin teaching naïve users basic computer techniques such as “drag anddrop” and other computer operations. See, for example, Sittig D F, JiangZ, Manfre S, et al., Evaluating a computer-based experiential learningsimulation: a case study using criterion-referenced testing, ComputNurs; 1995, Vol. 13, pages 17-24, incorporated herein by reference.

We have determined that the above described medical assessment processessuffer from two weaknesses: 1) test development requires re-generationof an examination with new material on a recurring (usually annual)basis; 2) although multiple choice questions demonstrate reliableperformance in measuring cognitive knowledge, the use of this format forassessing clinical problem solving has not been supported by theliterature. Another system was developed at the University of Wisconsin.This project served as the nidus for the Computer-Based Examination(CBX) developed by the National Board of Medical Examiners (NBME). See,for example, Friedman R B, A computer program for simulating thepatient-physician encounter, J Med Educ 1973, Vol. 48, pages 92-7;Clyman, Stephen G., Orr, Nancy A., Status Report on the NBME'sComputer-Based Testing, Academic Medicine 1990, Vol. 65, pages 235-41,incorporated herein by reference. NBME's CBX development project hasbeen in evolution for over a decade, and has demonstrated validity inexamining professional degree candidates. See, for example, Solomon D J,Osuch J R, Anderson K, et al., A pilot study of the relationship betweenexperts' ratings and scores generated by the NBME's computer-basedexamination system, Academic Medicine 1992, Vol. 67, pages 130-32,incorporated herein by reference.

However, we have determined that the CBX model suffers from the problemthat the clinical simulations are “hard-wired” in computer source codewhich must be re-coded for each new examination. Once the simulation hasbeen used widely, the examination contents are no longer secure,necessitating continuous cycles of new simulation development.

The expert system literature describes the evolution in evaluation andtraining systems. Early artificial intelligence/expert system workconcentrated on “rules of thumb” or heuristics to representproblem-solving strategies identified by domain experts. See, forexample, David J M, Krivine J P, Simmons R., Second generation expertsystems: a step forward in knowledge engineering, in: David J M, KrivineJ P, Simmons R. Second Generation Expert Systems, Springer Verlag, NewYork, N.Y. 1993, pages 3-23, incorporated herein by reference. We havedetermined that these rule-based systems were necessarily constrained tonarrow domains, and that the knowledge contained in the rules wasdifficult to validate. Id.

In addition, early expert systems suffered from rapidly decliningperformance when exposed to circumstances outside narrowly defineddomains. See, for example, Davis R. Expert systems: where are we andwhere do we go from here, AI Magazine, 1983, Vol. 3, pages 3-22; SimmonsR. Generate, Test and Debug: A paradigm for combining associational andcausal reasoning, in: David M, Krivine JP, Simmons R., Second GenerationExpert Systems, Springer Verlag, New York, N.Y. 1993, pages 79-92,incorporated herein by reference. We have determined that thisphenomenon occurred at least in part due to interactions among the manyrules needed to define a domain. Recent work indicates that therobustness of such systems is enhanced by providing knowledge ofdifferent types. See, for example, Simmons R, Davis R., The roles ofknowledge and representation in problem solving, In: David M, Krivine JP, Simmons R., Second Generation Expert Systems, Springer Verlag, NewYork, N.Y. 1993, pages 27-45, incorporated herein by reference.

We have further determined that experts generally not only relate to onedimension of knowledge when defining a rule, but also rely uponexpansive knowledge of how systems work (i.e., physiology andpathophysiology in the medical domain) in performing real-worldproblem-solving. See, for example, Davis R., Expert systems: where arewe and where do we go from here, AI Magazine, 1983, Vol. 3, pages 3-22,incorporated herein by reference. This realization has led tore-thinking regarding structure of knowledge-based systems to reflectthe tasks such a system should accomplish, the methods the system shoulduse to accomplish the tasks, and the knowledge required to support thesemethods. See, for example, David J M, Krivine J P, Simmons R., Secondgeneration expert systems: a step forward in knowledge engineering, In:David J M, Krivine J P, Simmons R., Second Generation Expert Systems,Springer Verlag, New York, N.Y. 1993, pages 3-23, incorporated herein byreference.

We have also determined that knowledge-acquisition for such systemsentails development of a model for the domain and instantiation (i.e.,encode and enter needed information into the system's data structure) ofthe model with information acquired from knowledge donors. See, forexample, David M, Krivine J P, Simmons R., Second generation expertsystems: a step forward in knowledge engineering, In: David M, Krivine JP, Simmons R., Second Generation Expert Systems, Springer Verlag, NewYork, N.Y. 1993, pages 3-23; Breuker J, Weilenga B., Models of expertisein knowledge acquisition, In: Gida and Tasso (eds), Topics in ExpertSystem Design: Methodologies and Tools, North Holland Publishing, 1989,incorporated herein by reference.

To obviate the above described weaknesses, we have determined that it isdesirable to provide a computer-based testing project which will: 1)instantiate medical knowledge as object-oriented data structures knownas knowledge base of family medicine; 2) utilize the medical knowledgestructures to create realistic clinical scenarios (simulated patients);and 3) assess the candidate's clinical problem solving ability as theeffective intervention in the clinical progress of the simulated patientthrough the selection of various actions made available by the testingsystem.

Applicants have recognized a need for a method and system for evaluatingor educating a user using belief networks or causal probabilisticnetworks, such as Bayesian networks, to describe health state evolution,medical finding reveal structures, and/or management plan critiques.

Applicants have also recognized a need for an expert system forfacilitating a user in the treatment of an actual patient using beliefnetworks or causal probabilistic networks, such as Bayesian networks, todescribe health state evolution, medical finding reveal structures,and/or management plan critiques.

SUMMARY OF THE INVENTION

The computer-based testing system described herein represents knowledgeat multiple levels of complexity. For example, reactive airways diseaseis represented as a series of health states: Normal (Non-reactive)Airways, Reactive Airways-Mild, Reactive Airways-Moderate, and ReactiveAirways-Severe. Each health state contains identifiers which relate theparticular health state to precedents and antecedents (e.g., NormalAirways serves as the precursor health state for Mild Reactive airwaysdisease, and Mild, Moderate and Severe Reactive Airways Diseaserepresent target health states from the Normal circumstance.)

Each health state in turn has associated findings, and specificfindings. For example, the Normal Airways state, the Finding “Shortnessof Breath” is instantiated with the Specific Finding “No shortness ofbreath.” Similarly, other Findings such as Respiratory Function andSevere Asthma Attack Frequency are instantiated with correspondingnormal Specific Findings (Normal Respiratory Functions, and No SevereAttacks.) This representation transports to each new health state in amanner which we have determined to be analogous to diagnosis. See, forexample, Genesereth M., Diagnosis using hierarchical design models,Proc. National Conference on AI, 1982, incorporated herein by reference.

The computer-based testing system of the present invention partitionsknowledge into fundamental types: Health States, Agents, Findings,Specific Findings and Patterns describe system behaviors andcharacteristics. Courses-of-Action describe human activities whichmodify and evaluate the health state information and characteristicsdescribed in the model. Subdivision of knowledge types in this mannerfacilitates the knowledge acquisition process. This subdivision alsopromotes multiple levels of knowledge abstraction, which enhances thesystem's ability to represent varying levels of complexity.

For example, in the Computer-Based Testing System, a pattern such asincidence is further sub-divided into sub-patterns such as incidence infemales versus males, and incidence in various racial/ethnic groups.

Multiple levels of abstraction and types of knowledge impose asubstantial knowledge acquisition challenge. Knowledge acquisitionincludes several possible methodologies, including direct questioning ofdomain experts/protocol analysis, see, for example, Ericsson KaA, SimonH A, Protocol Analysis: Verbal Reports as Data, MIT Press. Cambridge,Mass. 1984, incorporated herein by reference, psychometric methods, see,for example, Kelly G A, The Psychology of Personal Constructs, NortonPress, New York, N.Y. 1955, incorporated herein by reference, andethnographic methods, Suchman L A, Trigg R H, Understanding Practice:Video as a Medium for Reflection and Design, In: Greenbaum, J, Kyng M(eds)., Design at Work: Cooperative Design of Compute Systems, LawrenceEarlbaum Associates 1991, pages 65-89, incorporated herein by reference.

Advantageously, the Computer-Based Testing System of the presentinvention has included a blend of these approaches. Direct questioninghas been used in querying family practice physicians regarding theirknowledge of and approaches to specific knowledge domains (such asosteoarthritis). Additionally, knowledge acquisition has included accessto appropriate scientific literature, which functionally serves toprovide an ethnographic assay of actual practice. Knowledge acquisitionhas also entailed protocol analysis, both in terms of analyzing specificphysicians' problem solving methodologies and incorporating explicitclinical processes such as those presented in published clinicalguidelines (a specific example here is the otitis media with effusionguideline developed by the Agency for Health Care Policy and Research).

To facilitate development of such a system, the present invention isdivided into three components: the knowledge base, the patientsimulation generator, and the presentation system. The knowledge basehas been designed and represented as a series of entity-relationships.The model has several fundamental entities: Patient, Health States,Findings, Courses of Action, and Agents. These entities haverelationships of INTERACTS_WITH, CONTACTS, IS_RELATED, EXHIBITS, HAS,EXPOSED_TO, LEADS_TO, ASSOC_WITH, LINKS_TO, USES, IDENTIFY, MANAGE,ALTER, REVEAL, and EVALUATE.

FIG. 1 describes an overall or conceptual view of the entities andrelationships included in the model. Rectangles indicate entitiesbetween entities in the model. Hexagons indicate relationships. Solidlines indicate Medical Knowledge Relationships (e.g., a course of actionsuch as treatment with non-steroidal anti-inflammatory agents can modifyspecific findings such as pain in the patient with osteoarthritis.)Dotted lines indicate Simulation/Evolution relationships which definehow a particular domain simulation has proceeded.

The patient simulation generator of the present invention relies upon aseries of generation methods to instantiate patients for presentation tothe certification/recertification candidate. The processes function toevolve the patient forward (to reflect progression of the diseaseprocess and response to interventions) and backward in time (to create apast history for the patient.) To accomplish these tasks, the systemutilizes processes for:

-   1. Content specification—these processes define the scope of the    simulation-   2. Patient generation:    -   Past History (“backward” generation)    -   Present and Future History (“forward” generation)-   3. Simulation processes (in addition to patient generation):    -   Interface processes (for presentation of the patient findings        developed from generation processes.)    -   Book-keeping processes (for keeping track of candidates'        responses and patient evolution)

The patient generation process proceeds on the basis of a specifichealth state identifier (coded in the database as a name and SNOMEDcode) passed to the process at the start of the simulation. The SNOMEDInternational structured vocabulary is a versatile nomenclature fordescribing medical ideas. See, for example, Cote R A, Rothwell D J,Palotay J L, Beckett R S, Brochu L, editors, SNOMED International: Thesystematized nomenclature of human and veterinary medicine, 3rd ed.Northfield, Ill., College of American Pathologists, 1993, incorporatedherein by reference. This nomenclature allows one to make inferencesfrom the codes used to represent each idea. For instance, the codeF-37022 represents “retrosternal chest pain.” The first character, “F,”indicates that the code is from a broad class of ideas called functions.The next to digits, “37,” indicate that the code involves a refinementof the code F-37000, “chest pain, not otherwise specified.” Similarly,code F-37020 specifies “precordial chest pain.” The code F-37022 impliesthat retrosternal chest pain is a kind of precordial chest pain, whichis a kind of chest pain, which is a kind of function.

The generation process produces a complete patient description whichreflects the EXHIBITS, HAS, INTERACTS-WITH, EXPOSED-TO, IS-RELATED, andCONTACTS relationships described earlier. These generated entityrelations are stored as a collection of records referred to as the“White Board” data structures. The information in these records servesas input to the patient evolution process, which in turn evolves thepatient's health status and medical/personal characteristics as afunction of the passage of time or physician/examinee intervention.

The original patient generation process is generally called once at thesession's start; the system calls the evolution processes repeatedly inresponse to time progression and physician action.

The first phase of patient generation entails development of thepatient's history outline. This outline describes the series of healthstates and risk factors the patient experienced to reach the currenthealth state, TS. To develop TS, the system first calls the procedureGenderRace, which establishes the patient's sex and racial/ethnicorigin. Next, the system establishes the patient's age and age at onsetthrough the OnsetAge procedure. The CreatePerson process then assignsthe patient a birth date and name.

Once the patient's age, sex, racial/ethnic origin, and age at onset ofthe condition have been established, the OutlineFirstStep proceduredefines the precursor states and risk factors which serve as thesubstrata for evolving the patient to the current time and target healthstate. The OutlineGeneralStep procedure is then called iteratively untilthe patient has arrived at the current TS. These processes are describedin greater detail below.

Logical and procedural knowledge in the database described as “reasoningelements” (RE) (for example, Bayesian network describing a generationmethod, Bayesian network describing a treatment plan, and the like),included in the generation methods described above, “shape selectors”which describe distributions for the n patterns by which health statesevolve (patterns in turn are specified by findings and subpatterns), andcourses of action (COA) which represent possible further diagnostic andmanagement strategies which candidates might select.

The patterns and subpatterns are represented as probabilitydistributions (discrete and continuous as appropriate for particularfinding) specified through the knowledge acquisition process. At thebeginning of a simulation, random number generation is used to select a“master percentile” (MP) which then serves as the reference forselecting particular patterns, findings and subpatterns from theappropriate specified distributions. These selected patterns are queriedto provide description of specific findings such as hyperglycemia inresponse to physician/examinee requests for information which are in theform of “courses of action” for a particular health state (e.g.,hyperglycemia as a manifestation of diabetes.)

Once presented with the patient description (age, race, sex, clinicalfindings), the candidate then selects appropriate COA's for furtherevaluation and/or management of the patient's health state. Selection ofan interventional COA invokes pattern modifiers which evolve thepatient's health state by implementing shape modifiers. These modifiersact upon the initially selected health state patterns to redefine thepatient's health state or findings (e.g., a COA of insulinadministration would alter the hyperglycemic finding specified in thehealth state descriptions for diabetes mellitus.)

As mentioned earlier, COA's also include options for furthertesting/diagnostic procedures. For example, the candidate might chooseto select a glycosylated hemoglobin evaluation; the COA process wouldaccess the pattern for glycosylated hemoglobin instantiated at thebeginning of the simulation but which might not be reported unlessspecifically asked for by the candidate.

A COA can modify the health state in which a patient exists at one pointin time. When the candidate selects such a COA, the simulated patientevolves to a new health state patterns associated with the new healthstate in the knowledge base. In order to avoid “state explosion”, healthstates closely associated with each other are represented as parallelhealth states not as combined health state entities.

For example, the initially generated patient for a case ofosteoarthritis might demonstrate mild osteoarthritis. However, otherhealth states, such as obesity, might influence the progress of thepatient's arthritis from mild to moderate or severe disease. To avoidcombinatoric health state explosion, we have implemented a concept ofparallel networks of health states. In this representation, anewly-generated patient will exhibit instantiated health state patternsfor the primary domain (in this case osteoarthritis) and for theparallel health states (obesity in this example) which influence theprimary health state's progress.

As shown in FIG. 2, osteoarthritis can progress over time from thenormal state to mild, moderate or severe osteoarthritis. For thisparticular illness, progress occurs in one direction only;osteoarthritis does not regress once developed, but can stabilize at aparticular degree of severity. Obesity represents a parallel healthstate which can influence the progression of osteoarthritis. Mild,moderate, and severe obesity can influence this progress at differentrates: the model permits representation of greater impact for moresevere obesity states. Notice also that obesity can regress (e.g.,severe obesity can revert to moderate obesity, etc.).

Any one of a number of health states might exist which could progressindependently of osteoarthritis. For example, the patient who hasosteoarthritis will frequently utilize non-steroidal anti-inflammatorydrugs (NSAID's) for treatment. These agents can improve the symptoms ofosteoarthritis, but also impact on the parallel state of peptic ulcerdisease. Treatment with NSAID's can induce an ulcer, which can thenevolve either on the basis of physician/examinee intervention for it,and/or for the course and treatment for other parallel health states,and time with the course and treatment of osteoarthritis.

The computer based testing system's fidelity depends upon access to arich representation of health state-specific knowledge. This knowledgeconsists, as described above, in more detail below. The templateincludes a NAME for the health state and an associated SNOMED code. Thetemplate also includes specific descriptions of the FINDINGS, PATTERNSand SUBPATTERNS for these FINDINGS. The patterns and subpatterns arestored as a series of time and value pairs. As an example of suchpatterns, consider the example of Reactive Airways Disease (RAD). Onefinding of interest is the prevalence of RAD as a function of age, sex,and race. The prevalence for this finding appears in the knowledge baseas collection of graphs illustrating the population prevalenceconditioned on age, sex and race. Likewise, data such as acuteexacerbation rates are represented as event rate distributions. Thesubpatterns also include information describing how various treatmentmodalities will modify the exacerbation rate and other pertinentfindings such as peak expiratory flow rates and symptoms such asshortness of breath.

The present invention provides a prototypical process for developingdomain-specific knowledge. The template for each domain includes, forexample, the following hierarchy:

-   HEALTH STATE: {name assigned by the knowledge donor, e.g., “Normal    Airway Reactivity”}-   SNOMED CODE: {appropriate SNOMED code}-   PREVALENCE: {age-sex-race specific prevalence; represented as    pattern}-   INCIDENCE: {age-sex-race specific incidence; represented as pattern}-   FINDING: {general name for set of findings, e.g., “Asthma Attack    Frequency” in reactive airways disease}-   Specific Finding: {description of specific instance of a FINDING;    e.g., for the FINDING of asthma attack frequency, one specific    finding is “No Attacks”, associated with “Normal Airway Reactivity”)

Each HEALTH STATE affects multiple FINDINGS, which in turn have SpecificFindings appropriate for that FINDING in that HEALTH STATE. Data such asincidence, prevalence, and attack rates are represented as PATTERNS(graphical functions which support the patient generation simulationprocesses). The information is collected in paper template form, andthen transferred into computer-readable format using, for example, anystandard Knowledge Acquisition (KA) tool to enter the information intoan object-oriented database.

The KA “front end” may be developed, for example, in the Visual Basic®and Visual C++® programming environments. Courses-of-Action (COA), suchas further evaluation and/or management strategies, are entered using astandard editor that creates text files describing appropriateevaluation/management steps to support the simulation processes. The COAeditor may also be designed under the Microsoft Visual environmentsmentioned earlier.

The knowledge acquisition step includes the following subcomponents:

-   A. Health state specification-   B. Enumeration of FINDINGs for the health state, and agreement among    the development team members-   C. Population of templates with knowledge-   D. Entry of health state knowledge into knowledge base using KA tool    and/or direct high level pseudo-coding-   E. Debugging, including generating multiple simulations, to test    system stability/credibility-   F. Validation including review of generated cases by representative    groups of family physicians

It is a feature and advantage of the present invention to: (1) allowtesting at remote sites and convenient times; (2) uniformly test anexpanded range of important family practice activities, with fewerquestions on exotic problems; (3) adapt tests to examinees' responses orneeds; and (4) create reasonable questions at test sites to simplifyadministrative, economic, and especially security issues.

It is another feature and advantage of the present invention to providean approach that does not incur high maintenance costs and producesefficient and affordable scenarios for a computer-based testing system.

It is another feature and advantage of the present invention to providea formal model of family medicine to achieve a relevant and realisticimplementation of a computer based examination.

It is another feature and advantage of the present invention to providean examination that does not require replacement with new questions inorder to preserve security of the certification process.

It is another feature and advantage of the present invention to providea computer based testing system that may measure problem-solvingcapabilities.

It is another feature and advantage of the present invention to providea computer based testing system that relies upon a knowledge base offamily practice which contains “patterns” and “subpatterns” which depictin probabilistic terms disease/condition incidence, prevalence,evolution over time, and response to interventions.

The present invention is based, in part, on our discovery that priorcomputer based testing systems suffer from various problems, includingthe problem that the clinical simulations are “hard-wired” in computersource code or static data structures which must be re-coded orreinstantiated for each new examination. Accordingly, in prior artcomputer based testing systems, once the simulation has been usedwidely, the examination contents are no longer secure, necessitatingcontinuous cycles of new simulation development.

The present invention is also based, in part, on our realization thatthe computer based testing system needs to be capable of efficientlygenerating new patient cases for each candidate examined, and capable ofeffectively testing a candidate's problem-solving ability. We havediscovered that the above may be accomplished using a knowledge base offamily practice which contains “patterns” and “subpatterns” which depictin probabilistic terms disease/condition incidence, prevalence,evolution over time, and response to interventions.

To achieve the above features and advantages, as well as other featuresand advantages that will be apparent from the detailed descriptionprovided below, a computer implemented simulation and evaluation methodsimulates interventions to a patient by a user, and evaluates theinterventions responsive to predetermined criteria and theinterventions. The method includes defining a test area to evaluate theuser on at least one predetermined criterion, selecting geneticinformation of the patient responsive to the test area, and generating apatient history responsive to the test area and the genetic information.The method also includes receiving at least one intervention input bythe user, and evaluating the user responsive to the intervention andpredetermined criteria.

In accordance with another embodiment of the invention, a computersystem and computer readable tangible medium is provided that stores theprocess thereon, for execution by the computer.

In accordance with another embodiment of the invention, a computerreadable tangible medium is provided that stores an object including theentity relationship model thereon, for execution by the computer.

It is another feature and advantage of another embodiment of the instantinvention to include a method for evaluating or educating a user. Themethod includes the following sequential, non-sequential, orsequence-independent steps. Plurality of parallel health state networksare generated, for example, by a user or a computer. One or more firstBayesian networks, which describe each of the parallel health statenetworks generated by a user or a computer. One or more second Bayesiannetworks, which describe rates of progression within and/or between theparallel health state networks, and describe task factors that affectthe rates of progression, generated by a user or a computer. One or morethird Bayesian networks which support reveal structures to limit displayof patient test data to patient test data specifically requested by theuser, are generated by a user or a computer. One or more fourth Bayesiannetworks which support plan critiques of queries of and treatmentprescribed by the user, are generated by a user or a computer.

A knowledge base is scripted by the computer from the one or more firstBayesian networks and the one or more second Bayesian networks. A modelpatient, at least in part, is instantiated by the computer from thescripted knowledge base. A course of action or a query for a specificmedical finding concerning the model patient is received by the computerfrom the user responsive to the instantiated model patient. If the queryis received, the specific medical finding is displayed by the computerto the user based at least in part on the one or more third Bayesiannetworks, and repeating the receiving step.

The model patient is evolved by the computer in accordance with theparallel health state networks and responsive to the received course ofaction. The receiving, displaying, and evolving steps are repeated bythe computer until the user has completed treatment of the modelpatient. An optimum combination of treatment and queries based, at leastin part, on the one or more fourth Bayesian networks and theinstantiated model patient is generated by the computer. The query andthe treatment by the user is evaluated by the computer in comparison tothe generated optimum combination of treatment and queries.

Optionally, the parallel health state networks describe primary networksdefining disease evolutions, secondary networks defining risk factorsaffecting progression through a particular or given primary network ofthe plurality of primary networks, and/or tertiary networks definingcausal probabilistic medical complications attributed to one or morestages in the primary network and/or medical complications attributed tomanagement of the one or more stages.

It is another feature and advantage of the instant invention to providea computer readable medium including instructions being executed by acomputer. The instructions instruct the computer to execute aneducational or testing system for physicians. The instructions includethe following sequential, non-sequential, or sequence-independent steps.One or more first belief networks which describes parallel health statenetworks are accessed, for example, by a computer. A knowledge base, atleast in part, is scripted from the one or more first belief networks bythe computer. A model patient, at least in part, is instantiated by thecomputer from the scripted knowledge base. Optionally, for the computerreadable medium, the parallel health state networks describe primarynetworks defining disease evolutions, secondary networks defining riskfactors affecting progression through a primary network of the pluralityof primary networks, and/or tertiary networks defining causalprobabilistic medical complications attributed to at least one stage inthe primary network and/or medical complications attributed tomanagement of the one or more stages.

Optionally, the instructions further include one or more second beliefnetworks, which describe rates of progression within and/or between theparallel health state networks, and describe task factors that affectthe rates of progression, are accessed by the computer or the user.Optionally, one or more third belief network, which supports revealstructures to limit display of patient test data to patient test dataspecifically requested by the user, are accessed, for example, by thecomputer. Optionally, one or more fourth belief networks which supportplan critiques of queries of and treatment prescribed by the user, areaccessed by the user or the computer. Optionally, the scripting stepincludes scripting the knowledge base by the computer, at least in part,from the one or more second belief networks. Optionally, a course ofaction or a query for a specific medical finding concerning the modelpatient is received by the computer from the user responsive to theinstantiated model patient. If the query is received, the specificmedical finding is displayed by the processor to the user based at leastin part on the at least one third network, and the receiving step isrepeated by the processor.

Optionally, the model patient is evolved by the computer in accordancewith the parallel health state networks and responsive to the receivedcourse of action. Optionally, the receiving, displaying, and evolvingsteps are repeated by the computer until the user has completedtreatment of the model patient. Optionally, an optimum combination oftreatment and queries is generated by a processor based on the one ormore fourth belief networks and the instantiated model patient.Optionally, the query and the treatment by the user are evaluated by thecomputer in comparison to the generated optimum combination of treatmentand queries.

It is another feature and advantage of the instant invention to includea system for evaluating or educating a user. The system includes meansfor scripting a knowledge base from at least one first belief networkand at least one second belief network. The system includes means forinstantiating a model patient, at least in part, from the scriptedknowledge base. The system includes means for receiving a course ofaction or a query for a specific medical finding concerning the modelpatient from the user responsive to the instantiated model patient. Thesystem includes means for displaying, if the query is received, thespecific medical finding to the user based at least in part on at leastone third belief network, and activating the receiving means.

The system includes means for evolving the model patient in accordancewith the at least one first belief network and the at least one secondbelief network and responsive to the received course of action.Optionally, the system includes means for communicating with thereceiving means, the displaying means, and the evolving means until theuser has completed treatment of the model patient.

Optionally, the system includes means for generating an optimumcombination of treatment and queries based on at least one fourth beliefnetwork and the instantiated model patient. Optionally, the systemincludes means for evaluating the query and the treatment by the user incomparison to the generated optimum combination of treatment andqueries.

Optionally, the system includes means for generating parallel healthstate networks describing primary networks defining disease evolutions,secondary networks defining risk factors affecting progression through aprimary network of the plurality of primary networks, and/or tertiarynetworks defining causal probabilistic medical complications attributedto at least one stage in the primary network and/or medicalcomplications attributed to management of the at least one stage.

Optionally, the system includes means for generating the at least onefirst belief network which describes each of the plurality of parallelhealth state networks. Optionally, the system includes means forgenerating the at least one second belief network which describes ratesof progression within and/or between the parallel health state networks,and describes task factors that affect the rates of progression.Optionally, the system includes means for generating the at least onethird belief network which support reveal structures to limit display ofpatient test data to patient test data specifically requested by theuser. Optionally, the system includes means for generating the at leastone fourth belief network which supports plan critiques of queries ofand treatment prescribed by the user.

It is another feature and advantage of the instant invention to includean expert system. The expert system includes a processor. The expertsystem also includes a computer-readable medium storing instructionsexecutable by the processor.

The instructions include the following sequential, non-sequential, orsequence-independent steps. Parallel health state networks are accessedby the processor and which describe primary networks defining diseaseevolutions, secondary networks defining risk factors affectingprogression through a primary network of the plurality of primarynetworks, and/or tertiary networks defining causal probabilistic medicalcomplications attributed to at least one stage in the primary networkand/or medical complications attributed to management of the at leastone stage.

One or more first belief networks, which describe each of the pluralityof parallel health state networks, are accessed by the processor. One ormore second belief networks, which describe rates of progression withinand/or between the parallel health state networks, and describe taskfactors that affect the rates of progression, are accessed by theprocessor. One or more third belief networks, which support revealstructures to limit display of patient test data to patient test dataspecifically requested by the user, are accessed by the processor. Oneor more fourth belief networks, which support plan critiques of queriesof and treatment prescribed by the user, are accessed by the processor.Patient data for an actual patient is received by the processor by userinput.

A virtual patient having characteristics consistent with the receivedpatient data and based, at least in part, on the at least one firstbelief network and the at least one second belief network isinstantiated by the processor. A query for a specific medical findingconcerning the actual patient, or a course of action responsive to atleast one normal or abnormal health state of the plurality of healthstates of the virtual patient is generated by the processor. The normalor abnormal health state corresponds to at least part of the receivedpatient data. The specific medical finding from the user, if a querytherefor is generated.

The virtual patient is evolved by the processor in accordance with theat least one first belief network and/or the at least one second beliefnetwork, and responsive to the received specific medical finding and/orthe generated course of action. Optionally, the instructions for theexpert system further includes repeating the generating, receiving, andevolving instruction steps until the user has dispensed treatment of theactual patient based on the generating course of action. Optionally, theinstructions further include storing the evolved virtual patient forsubsequent access by the user, and repeating the generating, receiving,evolving, repeating, and storing instruction steps upon each subsequentaccess by the user at least until the treatment of the actual patient iscompleted.

It is another feature and advantage of the instant invention to includea system for educating or evaluating a user. The system includes a modelpatient generator including a knowledge base scripted from one or morefirst causal probability networks, one or more second causal probabilitynetworks. The one or more first causal probability networks describeeach parallel health state network of a plurality of parallel healthstate network. The one or more second causal probability networksdescribe at least one rate of progression within and/or between theparallel health state networks, and which describe at least one taskfactor that affects the one rate of progression. The patient generatorinstantiates, upon user input, a model patient in a whiteboard, at leastin part, from the scripted knowledge base. The patient generatorreceives a course of action or a query for a specific medical findingconcerning the model patient from the user responsive to theinstantiated model patient. The whiteboard optionally displays, if thequery is received, the specific medical finding to the user based, atleast in part, on at least one third belief network, which supportspatient health state reveal structures. The whiteboard evolves the modelpatient in accordance with the plurality of parallel health statenetworks and responsive to the received course of action.

It is another feature and advantage of the instant invention to includea system communicatable with a computer network. The system includes aserver communicatable with a user via the computer network. The serveris in communication with a processor and a computer-readable mediumstoring instructions executable by the processor. The instructionsinclude the following sequential, non-sequential, orsequence-independent instruction steps. Parallel health state networksare accessed by a user or the processor and which describes primarynetworks defining disease evolutions, secondary networks defining riskfactors affecting progression through a primary network of the pluralityof primary networks, and/or tertiary networks defining causalprobabilistic medical complications attributed to one or more stage inthe primary network and/or medical complications attributed tomanagement of the one or more stage.

One or more first belief networks, which describe each of the pluralityof parallel health state networks, are accessed by the user or theprocessor. One or more second belief networks which describe rates ofprogression within and/or between said plurality of parallel healthstate networks, and to describe task factors that affect the rates ofprogression, are accessed by the user or the processor. One or morethird belief networks, which supports plan critiques of queries of andtreatment prescribed by the user, are accessed by the user or theprocessor.

Patient data for an actual patient are received by user input to theprocessor. A virtual patient, having characteristics consistent with thereceived patient data and based, at least in part, on the one or morefirst belief networks and the one or more second belief networks, isinstantiated by the processor. A query to the user for a specificmedical finding concerning the actual patient, or a course of actionbased, at least in part on the virtual patient and the one or more thirdbelief networks are generated by the processor. The specific medicalfinding is received by the processor from the user responsive to thegenerated query. The virtual patient is evolved by the processor inaccordance with the one or more first belief network and/or the one ormore second belief network, and responsive to the received specificmedical finding.

Optionally, the instructions of expert system further include repeatingthe generating, receiving, and evolving instructions steps until theuser has dispensed treatment of the actual patient based on thegenerating course of action. Optionally, the evolved virtual patient isstored by the processor for subsequent access by the user. Optionally,the generating, receiving, evolving, repeating, and, storinginstructions are repeated by the computer upon each said subsequentaccess by the user at least until the treatment of the actual patient iscompleted.

It is another feature and advantage of the instant invention to includea system communicatable with a computer network. The system includes aserver communicatable with a user via the computer network. The serveris in communication with a processor and a computer-readable mediumstoring instructions executable by the processor.

The instructions include the following sequential, non-sequential, orsequence-independent instruction steps. Optionally, parallel healthstate networks are accessed by the processor or a user and whichdescribe primary networks defining disease evolutions, secondarynetworks defining risk factors affecting progression through a primarynetwork of the primary networks, and/or tertiary networks definingcausal probabilistic medical complications attributed to one or morestages in the primary network and/or medical complications attributed tomanagement of the one or more stages.

One or more first belief networks which describe each of the pluralityof parallel health state networks are accessed. One or more secondbelief networks, which describe rates of progression within and/orbetween said plurality of parallel health state networks, and describetask factors that affect the rates of progression, are accessed. One ormore third belief networks, which support reveal structures to limitdisplay of patient test data to patient test data specifically requestedby the user, are accessed. One or more fourth belief networks, whichsupports plan critiques of queries of and treatment prescribed by theuser, are accessed. A knowledge base is scripted by the processor or auser from the one or more first belief networks and the one or moresecond belief networks.

A model patient is instantiated by the processor, based, at least inpart, from the scripted knowledge base. A course of action or a queryfor a specific medical finding concerning the model patient is receivedby the processor from the user responsive to the instantiated modelpatient. If the query is received by the processor, the specific medicalfinding is displayed by the processor to the user based at least in parton the one or more third belief networks, and repeating the receivinginstruction. The model patient is evolved by the processor in accordancewith at least one of the one or more first belief networks and the oneor more second belief networks and responsive to the received course ofaction.

Optionally, the instructions further include repeating by the processorthe receiving, displaying, and evolving instructions until the user hascompleted treatment of the model patient. Optionally, an optimumcombination of treatment and queries is generated by the processor basedon the one or more fourth belief networks and the instantiated modelpatient. Optionally, the query and the treatment by the user isevaluated by the processor in comparison to the generated optimumcombination of treatment and queries.

It is a feature and advantage of another embodiment of the instantinvention to include a knowledge base module for an educational ortesting system or an expert system. The knowledge base module includesone or more first causal probability networks, which describe eachparallel health state network of a plurality of parallel health statenetworks. The module includes one or more second causal probabilitynetworks, which describe one or more rates of progression within and/orbetween the parallel health state networks, and which describe one ormore task factors that affect the one or more rates of progression. Themodule also includes one or more third causal probability networks,which describe plan critiques including peer-accepted courses of actionfor addressing the parallel health state networks.

It is a feature and advantage of another embodiment of the instantinvention to include a computer network appliance. The network applianceincludes a thin client programmably connected via a computer network toa single web hosting facility. The single web hosting facility includesa server communicatable with a user via the computer network. The serveris in communication with a processor and a computer-readable mediumstoring instructions executable by the processor.

The instructions include the following sequential, non-sequential, orsequence-independent instruction steps. Parallel health state networksdescribing primary networks defining disease evolutions, secondarynetworks defining risk factors affecting progression through a primarynetwork of the plurality of primary networks, and/or tertiary networksdefining causal probabilistic medical complications attributed to atleast one stage in the primary network and/or medical complicationsattributed to management of the at least one stage are accessed by theuser or the processor.

One or more first belief networks, which describe each of the parallelhealth state network, are accessed by the user or the processor. One ormore second belief networks, which describe rates of progression withinand/or between said plurality of parallel health state networks, anddescribe task factors that affect the rates of progression, are accessedby the user or the processor. One or more third belief networks, whichsupport reveal structures to limit display of patient test data topatient test data specifically requested by the user, are accessed bythe user or the processor. One or more fourth belief networks, whichsupports plan critiques of queries of and treatment prescribed by theuser, are accessed by the user or the processor.

A knowledge base from the one or more first belief networks and/or theone or more second belief networks is scripted by the processor. A modelpatient, at least in part, is instantiated from the scripted knowledgebase by the processor. A course of action or a query for a specificmedical finding concerning the model patient is received by theprocessor from the user responsive to the instantiated model patient. Ifthe query is received, the specific medical finding is displayed by theprocessor to the user based in part on the one or more third beliefnetworks, and repeating the receiving instruction step. The modelpatient is evolved by the processor in accordance with the parallelhealth state networks and responsive to the received course of action.

Optionally, the instructions for the computer network appliancerepeating the receiving, displaying, and evolving instruction stepsuntil the user has completed treatment of the model patient. Optionally,the instructions include generating an optimum combination of treatmentand queries based on the one or more fourth belief networks and theinstantiated model patient, and evaluating the query and the treatmentby the user in comparison to the generated optimum combination oftreatment and queries.

It is a feature and advantage of another embodiment of the instantinvention to include a computer network appliance. The computer networkappliance includes a thin client programmably connected via a computernetwork to a single web hosting facility. The single web hostingfacility includes a server communicatable with a user via the computernetwork. The server is in communication with a processor and acomputer-readable medium storing instructions executable by theprecessor.

The instructions include the following sequential, non-sequential, orsequence-independent instruction steps. Parallel health state networksdescribing primary networks defining disease evolutions, secondarynetworks defining risk factors affecting progression through a primarynetwork of the plurality of primary networks, and/or tertiary networksdefining causal probabilistic medical complications attributed to atleast one stage in the primary network and/or medical complicationsattributed to management of the at least one stage, are accessed by theuser or the process. One or more first belief networks, which describeseach of the parallel health state networks, are accessed by the user orthe processor. One or more second belief networks, which describe ratesof progression within and/or between said plurality of parallel healthstate networks, and describe task factors that affect the rates ofprogression, are accessed by the user or the processor. One or morethird belief networks which support plan critiques of queries of andtreatment prescribed by the user, are accessed by the user or theprocessor.

Patient data for an actual patient are received by user input. A virtualpatient having characteristics consistent with the received patient dataand based, at least in part, on the one or more first belief networksand/or the one or more second belief networks are instantiated by theprocessor.

A query to the user for a specific medical finding concerning the actualpatient, or a course of action based, at least in part on the virtualpatient and the one or more third belief networks are generated by theprocessor. The specific medical finding is received by the processorfrom the user responsive to the generated query. The virtual patient isevolved by the processor in accordance with the one or more first beliefnetworks and/or the one or more second belief networks, and responsiveto the received specific medical finding.

The instructions for the optionally expert system further includerepeating the generating, receiving, and evolving instruction stepsuntil the user has dispensed treatment of the actual patient based onthe generating course of action, and storing the evolved virtual patientby the processor for subsequent access by the user. Optionally, thegenerating, receiving, evolving, repeating, and storing instructionsteps are repeated by the processor upon each said subsequent access bythe user at least until the treatment of the actual patient iscompleted.

It is a feature and advantage of another embodiment according to theinstant invention to include a system communicatable with a computernetwork. The system includes a server communicatable with a user via thecomputer network. The server is in communication with a processor and acomputer-readable medium storing instructions executable by theprocessor.

The instructions include the following sequential, non-sequential, orsequence-independent steps. Parallel health state networks, describingprimary networks defining disease evolutions, secondary networksdefining risk factors affecting progression through a primary network ofthe plurality of primary networks, and/or tertiary networks definingcausal probabilistic medical complications attributed to at least onestage in the primary network and/or medical complications attributed tomanagement of the at least one stage, are accessed by the processor or auser.

One or more first belief networks, which describe each of the pluralityof health states in parallel networks, are accessed by the processor ora user. One or more second belief networks, which describe transitionsbetween health states within parallel networks, and describe taskfactors that affect the rates of progression, are accessed by theprocessor or a user. One or more third belief networks, which supportreveal structures to limit display of patient test data to patient testdata specifically requested by the user, are accessed by the processoror a user. One or more fourth belief networks, which support plancritiques of queries of and treatment prescribed by the user, areaccessed by the processor or a user.

A knowledge engineer, such as a developer or a standard data miningprocess, scripts a knowledge base by specifying one or more first beliefnetworks and/or the one or more second belief networks. A model patientis instantiated by the processor based, at least in part, from thescripted knowledge base. A course of action or a query for a specificmedical finding concerning the model patient is received by theprocessor from the user responsive to the instantiated model patient. Ifthe query is received, the specific medical finding is displayed by theprocessor to the user based at least in part on the one or more thirdbelief networks, and repeating the receiving instruction step. The modelpatient is evolved by the processor in accordance with the one or morefirst belief networks and/or the one or more second belief networks andresponsive to the received course of action.

Optionally, the instructions further include repeating the generating,receiving, and evolving instruction steps until the user has dispensedtreatment of the actual patient, at least in part, based on thegenerated course of action, and storing the evolved virtual patient bythe processor for subsequent access by the user. Optionally, thegenerating, receiving, evolving, repeating, and storing instructionsteps are repeated by the processor upon each subsequent access by theuser at least until the treatment of the actual patient is completed.

It is a feature and advantage of another embodiment of the instantinvention to include a method for educating or evaluating a user. Themethod includes the following sequential, non-sequential, orsequence-independent steps. A virtual patient is instantiated fordisplay to the user, for example, by a computer. The virtual patientincludes a number of health states. A query is received from the userfor a medical finding concerning the instantiated virtual patient.Optionally, responsive to the received query, a specific medical findingis generated at least in part from a first causal probabilistic networkdefining a health state reveal structure corresponding to theinstantiated virtual patient. Optionally, responsive to the receivedquery, an indication of an inappropriate query is generated based, atleast in part, on a second causal probabilistic network defining amedical practice management plan. By way of illustration, the medicalpractice management plan includes healthcare provider or medicalinsurance-approved medical finding queries.

It is a feature and advantage of another embodiment of the instantinvention to include a method for educating or evaluating a user. Avirtual patient is instantiated by a computer for display to the user.The virtual patient includes a plurality of health stages. A query for amedical finding concerning the instantiated virtual patient is receivedby the computer. Responsive to the received query, a specific medicalfinding is generated by the computer, at least in part, from a firstcausal probability network defining a health state reveal structurecorresponding to the instantiated virtual patient. Responsive to thereceived query, an indication of an inappropriate query, based, at leastin part, on a second causal probability network defining a medicalpractice management plan. Responsive to the received course of action,an indication of an inappropriate course of action by the computerbased, at least in part, on the second causal probability network.Optionally, the medical practice management plan includes healthcareinsurer approved medical finding queries. Advantageously, such a methodis used to familiarize doctors new to a health care plan with managementapproved medical tests and/or medical procedures.

There has thus been outlined, rather broadly, the more importantfeatures of the invention in order that the detailed description thereofthat follows may be better understood, and in order that the presentcontribution to the art may be better appreciated. There are, of course,additional features of the invention that will be described hereinafterand which will form the subject matter of the claims appended hereto.

In this respect, before explaining at least one embodiment of theinvention in detail, it is to be understood that the invention is notlimited in its application to the details of construction and to thearrangements of the components set forth in the following description orillustrated in the drawings. The invention is capable of otherembodiments and of being practiced and carried out in various ways.Also, it is to be understood that the phraseology and terminologyemployed herein are for the purpose of description and should not beregarded as limiting.

As such, those skilled in the art will appreciate that the conception,upon which this disclosure is based, may readily be utilized as a basisfor the designing of other structures, methods and systems for carryingout the several purposes of the present invention. It is important,therefore, that the claims be regarded as including such equivalentconstructions insofar as they do not depart from the spirit and scope ofthe present invention.

Further, the purpose of the foregoing abstract is to enable the U.S.Patent and Trademark Office and the public generally, and especially thescientists, engineers and practitioners in the art who are not familiarwith patent or legal terms or phraseology, to determine quickly from acursory inspection the nature and essence of the technical disclosure ofthe application. The abstract is neither intended to define theinvention of the application, which is measured by the claims, nor is itintended to be limiting as to the scope of the invention in any way.

The above objects of the invention, together with other apparent objectsof the invention, along with the various features of novelty whichcharacterize the invention, are pointed out with particularity in theclaims annexed to and forming a part of this disclosure. For a betterunderstanding of the invention, its operating advantages and thespecific objects attained by its uses, reference should be had to theaccompanying drawings and descriptive matter forming a part hereof,wherein like numerals refer to like elements throughout, and in whichthere is illustrated preferred embodiments of the invention.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram describing an overall or conceptual view of theentities and relationships in the model used in the computer basedexamination system of the present invention;

FIG. 2 is a diagram describing the progression of osteoarthritis overtime from the normal state to mild, moderate or severe states ofosteoarthritis;

FIG. 3 is a detailed diagram of the family medicine model, including themajor entities, relations and modifying relations;

FIG. 4 is a flowchart of the overall process for the computer basedexamination system of the present invention;

FIG. 5 is a flowchart of the history outline process which generates thepatient history in the computer based examination system of the presentinvention;

FIG. 6 is a flowchart of the history generation process which findsvalues for the patient history in the computer based examination systemof the present invention;

FIG. 7 is a flowchart providing an overview of the stochastic process inaccordance with another embodiment of the computer based examinationsystem of the present invention;

FIG. 8 is a flowchart illustrating a first step in tracing previoushealth conditions to generate past medical history of the patient forthe stochastic process of the computer based examination system of thepresent invention;

FIG. 9 is a flowchart illustrating a second step in tracing previoushealth conditions to generate past medical history of the patient forthe stochastic process of the computer based examination system of thepresent invention;

FIG. 10 is an illustration of the entity-relationship model datastructure stored in the white board database when patients are notpre-generated;

FIG. 11 is an illustration of a modified entity-relationship model datastructure stored in the white board database when patients are notpre-generated;

FIG. 12 is an illustration of parallel network structures for thecomputer based examination system of the present invention;

FIGS. 13-14 are detailed flowcharts of the process of the computer basedexamination or assessment system of the present invention;

FIG. 15 is an illustration of a main central processing unit forimplementing the computer processing in accordance with a computerimplemented embodiment of the present invention;

FIG. 16 illustrates a block diagram of the internal hardware of thecomputer of FIG. 15;

FIG. 17 is a block diagram of the internal hardware of the computer ofFIG. 16 in accordance with a second embodiment;

FIG. 18 is an illustration of an exemplary memory medium which can beused with disk drives illustrated in FIGS. 15-17.

FIG. 19 is an illustration of a relational diagram of the Bayes networksand other supporting structures;

FIG. 20 is an illustration of an example using a Bayes network todescribe osteoarthritis;

FIG. 21 is an illustration of an example using a Bayes network togenerate a report when a user submits a medical finding query;

FIG. 22 is an illustration of examples of disease evolution described byparallel health state networks;

FIG. 23 is an illustration of an example of interactions betweenparallel health state networks;

FIG. 24 is an illustration of an example showing the relationshipsbetween entities in a health state;

FIG. 25 is an illustrative flow chart outlining operation of anembodiment of the instant invention;

FIG. 26 is an illustrative flow chart showing operation of anotherembodiment of the instant invention;

FIG. 27 is an illustrative flow chart showing operation of anotherembodiment of the instant invention; and

FIG. 28 is an illustration of a computer network architecture.

NOTATIONS AND NOMENCLATURE

The detailed descriptions which follow may be presented in terms ofprogram procedures executed on a computer or network of computers. Theseprocedural descriptions and representations are the means used by thoseskilled in the art to most effectively convey the substance of theirwork to others skilled in the art.

A procedure is here, and generally, conceived to be a self-consistentsequence of steps leading to a desired result. These steps are thoserequiring physical manipulations of physical quantities. Usually, thoughnot necessarily, these quantities take the form of electrical ormagnetic signals capable of being stored, transferred, combined,compared and otherwise manipulated. It proves convenient at times,principally for reasons of common usage, to refer to these signals asbits, values, elements, symbols, characters, terms, numbers, or thelike. It should be noted, however, that all of these and similar termsare to be associated with the appropriate physical quantities and aremerely convenient labels applied to these quantities.

Further, the manipulations performed are often referred to in terms,such as adding or comparing, which are commonly associated with mentaloperations performed by a human operator. No such capability of a humanoperator is necessary, or desirable in most cases, in any of theoperations described herein which form part of the present invention;the operations are machine operations. Useful machines for performingthe operation of the present invention include general purpose digitalcomputers or similar devices.

The present invention also relates to apparatus for performing theseoperations. This apparatus may be specially constructed for the requiredpurpose or it may comprise a general purpose computer as selectivelyactivated or reconfigured by a computer program stored in the computer.The procedures presented herein are not inherently related to aparticular computer or other apparatus. Various general purpose machinesmay be used with programs written in accordance with the teachingsherein, or it may prove more convenient to construct more specializedapparatus to perform the required method steps. The required structurefor a variety of these machines will appear from the description given.

BEST MODE FOR CARRYING OUT THE INVENTION

The computer-based testing system described herein represents knowledgeat multiple levels of complexity.

The computer-based testing system of the present invention partitionsknowledge into fundamental types: Health States, Agents, Findings,Specific Findings, Patterns and Sub-patterns describe system behaviorsand characteristics. Courses-of-Action describe tasks and methods usedto apply, modify, and evaluate the health state information andcharacteristics described in the model. Subdivision of knowledge typesin this manner facilitates the knowledge acquisition process. Thissubdivision also promotes multiple levels of knowledge abstraction,which enhances the system's ability to represent varying levels ofcomplexity.

For example, reactive airways disease is represented as a series ofhealth states: Normal (Non-reactive) Airways, Reactive Airways-Mild,Reactive Airways-Moderate, and Reactive Airways-Severe. Each healthstate contains identifiers which relate the particular health state toprecedents and antecedents (e.g., Normal Airways serves as the precursorhealth state for Mild Reactive airways disease, and Mild, Moderate andSevere Reactive Airways Disease represent target or sequential successorhealth states from the Normal circumstance.)

Each health state in turn has associated findings, and specificfindings. For example, in the Normal Airways state, “Asthma AttackFrequency” appears as a Finding which is instantiated with the SpecificFinding “No attacks.” Similarly, other Findings such as RespiratoryFunction and Severe Asthma Attach Frequency are instantiated withcorresponding normal Specific Findings (Normal Respiratory Functions,and No Severe Attacks.) This representation transports to each newhealth state in, what we have determined to be somewhat analogous todiagnosis.

Advantageously, the Computer-Based Testing System of the presentinvention in the knowledge acquisition process uses direct questioningin querying family practice physicians regarding their knowledge of andapproaches to specific knowledge domains (such as osteoarthritis).Additionally, knowledge acquisition has included access to appropriatescientific literature, which functionally serves to provide anethnographic assay of actual practice.

Overview of Testing/Recertification Process

The testing and/or recertification process, for example, unfolds asfollows. After initial certification, examinees initiate recertificationsoftware on workstations on computer systems. The examinee beginsrecertifying at any convenient time and could suspend the examination atthe conclusion of any simulated patient encounter. The software of thepresent invention presents a patient by using text, illustrations, stillpictures, and video. The examinee questions and examines the simulatedpatient, reaches conclusions about the situation, and suggests treatmentoptions. The simulated patient may express preferences about theseoptions.

After receiving a treatment plan, the patient leaves, maybe follows theplan, and perhaps later returns for follow-up. In the meantime, theexaminee sees other simulated patients. To discourage cheating, thesoftware offers so many cases that a diplomate observing anotherexaminee recertify gains little advantage with regard to test content.

The present invention maintains records of the information gathered, thehypotheses entertained, and the recommendations made for each patient.After monitoring performance on several similar cases (for instance,cases involving diagnosis and management of adult-onset diabetesmellitus), the program draws conclusions about the physician's abilityto handle this class of problems. If competence has been demonstrated,the class of problems may be removed from further consideration forseveral years. Until competence has been demonstrated, the physicianreceives feed-back on specific areas for improvement and continues tosee cases from this class of problems.

The testing and/or recertification process could eventually become acontinuous learning experience at the office or home. Somerecertification activities might qualify as continuing medicaleducation, partially offsetting the time needed to recertify. Examineescould anticipate failure to recertify and take corrective measures yearsbefore actually failing.

The present invention provides an approach that does not incur highmaintenance costs to maintain efficient and affordable examinations. Thepresent invention also provides a formal model of family medicine toachieve a relevant and realistic implementation of this kind ofcomputer-based examination.

In general, a model describes the kinds of information that could becollected regarding a topic. For instance, a model of a mailing addressshould include at least a name, street address, apartment number, city,state, and ZIP code. A database built upon this model could list theseitems for each entry. Not every item in the model should be describedfor every entry in the database; many addresses have no apartmentnumber. Incomplete database entries still provide useful information;even if a street address is missing, the city to search can be found.

Finally, the model limits what the database could do; it could noteasily list first names. A model of diagnostic medicine of the presentinvention includes diseases, historical and examination data, and linksbetween diseases and data. These models represent knowledge thatphysicians apply to uncertain or imprecise cases. The address examplesuggests a list of simple observations, called a database. A diagnosticprogram uses a collection of more abstract information, such as astatistical summary of a database, to draw inferences about a singlecase. The program and its information are often called a knowledge base.

We have determined that a well-designed formal model supportsautomatically created case simulations, reducing the long-term cost ofwriting cases by hand and improving security. The formal model of thepresent invention considers that medicine is full of diagnosticcomplexities including disease interaction. Thus, diabetes could changethe severity of pain experienced during an acute myocardial infarction.With this information, the knowledge base of the present invention isable to support a realistic simulation process—a simulated diabetichaving an acute myocardial infarction will experience a specificdiscomfort. The present invention attempts to carefully defineinteractions for a number of health states that constitute the bulk offamily medicine.

We have further determined that diagnosis and patient management areinextricably linked to time. Time receives relatively little attentionin many knowledge bases and is often summarized very succinctly. Forinstance, a knowledge base might describe “chest pain lasting more than30 minutes” as a symptom of acute myocardial infarction. This knowledgebase could misinterpret 29 minutes of chest pain as evidence againstacute myocardial infarction, and 2 years of chest pain as an indicatorof acute myocardial infarction. The present invention also supports therelated concepts of continuity of care and observation.

In addition to these problems, family physicians deal with a host ofissues that we have determined are not routinely modeled in diagnosticsoftware. Most of these issues reflect the overwhelming importance ofpatient management in family medicine.

First, family medicine occurs in a social context that is often ignoredin computer-generated simulations. Knowledge bases do not model socialinteractions or family structure.

Second, family practice patients arrive with attitudes shaped byexperience, and physicians must adjust their strategies to cope withthose attitudes. Adjustments range from changing interview style toaltering treatments. Variability in patient attitudes limits thelikelihood that there exists one best answer for groups of patients withsimilar medical conditions.

Third, family physicians emphasize helping patients improve the lengthand quality of their lives. Family physicians spend considerable timereassuring worried patients, alleviating symptoms, and preventing theonset or progression of disease.

We have determined that the final testing and/or recertificationproblem, evaluating the responses of diplomates, also requires a modelof what family physicians do. All dichotomous evaluations, especiallypass-fail tests, use arbitrary standards. The challenge is to setstandards using generally agreeable and meaningful criteria. The presentinvention provides the flexibility to determine to whom the criteriashould be agreeable—certainly to diplomates, but perhaps also topatients, insurers, or other customers. Specifying these customers willhelp establish meaningful criteria for certification decisions.

For instance, diplomates have an interest in maintaining respectedcredentials, patients want effective care, insurers desire low costs,and public health advocates have an interest in clinical guidelines. Itis not at all clear how to respond to these diverse interests. Thepresent invention delivers flexible models to describe the consequencesof family practice activities, as seen by various parties, so that boardcertification remains a pertinent process regardless of changes in thehealth care system.

We have determined that a model is needed to describe the scope offamily medicine in epidemiologic terms, while including the informationabout individual variation that differentiates individualized patientcare from public health. The model will be the foundation of a familypractice knowledge base storing data about family medicine. The modelalso supports other applications of benefit to family physicians.Specific software applications might involve medical records, structuredvocabularies, medical reference tools, decision support systems, andcontinuing education programs.

Data structures to describe the activities of family physicians includea series of entity-relation diagrams. In an entity-relation diagram,entities usually represent things (nouns). The relations (verbs)illustrate how the entities interact. For instance, an entity-relationdiagram of an address list might have an entity called “person,” and anentity called “place,” connected by a relation called “is at.” One couldread this diagram, “person is at place.” The person entity would storepeople's names, the place entity would store addresses, and the “is at”relation would describe when and why this person is at that place. Thus,a person could now live at one place, previously live at another place,and continuously work at the first place. One person, two places, andthree “is at” relations describe this address history. This addressmodel is flexible and realistic.

We have determined that an important class of events exist in the modelof family medicine, which we call “modifying relations,” or modifiers.In database terms, modifiers are relations between traditionalrelations. Modifiers extend the conventional entity relation diagram andprovide a means of managing statistically dependent events.

Model Structure

The family medicine model includes the major entities, relations andmodifying relations shown in detail FIG. 3. Formal concepts in the modelare capitalized throughout the text. The model emphasizes diagnostic andmanagement issues, variability in populations, and time. It describesconsequences of anatomic and physiologic processes, but largely omitsanatomic and physiologic reasoning as such. It is also capable ofdescribing interpersonal relationships and is expendable to include anexplicit representation of families or communities.

Modifiers (for example, Bayesian network from a Lead to relation,Bayesian network describing risk factors for progression, and the like)are relations that might change values in other relations. Dynamicentities and relations contain information relevant to patientsimulations. Dynamic information for an individual patient is derivedfrom data in other dynamic and static entities and relations. Thedynamic Record entity has relations mirroring the Population'srelations. Static entities and relations contain the best availablemedical knowledge, similar to data in medical literature.

The following major entities appear in the design: Populations, Records,Health States, Findings, Courses of Action, and Agents of Change.

Populations represent real humans; their relations should preciselydescribe all data that physicians consider. Populations can be largegroups with a shared characteristic, such as white males orsingle-parent families. An individual patient is a Population of 1; apregnant woman is a Population of 2; a nuclear family with 2 childrenand 2 parents is a Population of 4.

Records model beliefs about people; a Record's relations summarizeinferences about a Population. If a parent brings an infant to theoffice, this design represents the infant as a Population, the parent asanother Population, and the parent's description of the infant as aRecord. The physician can obtain historical information about the infantfrom two sources: the physician's medical Record of the infant, and theparent's Record of the infant. The physician can obtain currentobjective information by examining the infant as a Population. The datalinked to Populations are absolutely precise, but can be observed, if atall, only during medical encounters. Records summarize the history ofthose real data imprecisely and potentially inaccurately.

Populations have Records of themselves, modeling a patient's self-imageand memories. As with other Records, a patient's self-Record summarizeshistorical information with variable accuracy and might be thephysician's only source of some historical information.

A Population is primarily a list of relations with other entities. ARecord not only lists relations with other entities, but also definesencounters during which these relations were discovered. A Record cancontain conflicting data acquired at different encounters.

Health States include all normal health states; classic diseasepresentations; early, subtle, or late disease presentations; and somedisease combinations. Health States also include groups of Health Stateswith shared characteristics, such as cardiovascular diseases anddiseases of glucose intolerance. The SysteMetrics Corporation publishesDisease Staging Clinical Criteria, which define numerous stages in thedevelopment of diseases. See, for example, Gonella J S, Louis D Z, GozumM E, editors, Disease staging clinical criteria, 4th ed. Ann Arbor,Mich.: MEDSTAT Systems, 1994, incorporated herein by reference.

Each of these stages represents a distinct Health State entity in thisdesign. The SysteMetrics staging of diabetes mellitus defines stage 1.1as asymptomatic diabetes, stage 1.2 as symptomatic diabetes, stage 1.3as type I diabetes mellitus, and stage 2.1 as diabetes with end-organdamage. Each of these stages defines at least one Health State by thepresence of specific objective criteria.

Stage 2.1 might be divided into a group of Health States representingeach damaged end organ. To represent multiple end organ damage, onemight simply superimpose these states.

Findings include genetic, physiologic, symptomatic, physical, andtest-generated data, and clusters of such data. For instance,musculoskeletal chest pain could be a Finding. This Finding could be anexample of a Finding called chest pains, which would represent all kindsof chest pains. Chest pains could be an example of a still largerFinding called symptoms. Findings are defined by a collection of one ormore Features, whose current value can be described by a number on ascale. One Feature pertinent to pain is severity, which might bedescribed on a 10-point scale.

Structures called Patterns describe the possible values of each Featureover time. A Pattern typically lists a series of values andcorresponding percentiles at several points in time. Pediatric growthcharts are the most widely used real example of Patterns. A blank growthchart illustrates at least the following observations: (1) Normal birthweights vary within a narrow range. (2) Weight increases relativelyrapidly in the first few months and years. (3) The absolute variation inweight (e.g., the difference between 90th and 10th percentile weights)increases after birth. (4) Most people reach a fairly constant weight byearly adulthood. A pattern listing 10th and 90th percentile weights forpeople at age 0, 1 year, 2 years, and so on, illustrates the sameconcepts.

Growth charts also predict future values from past information. A childat the 50th percentile for weight now is expected to stay near the 50thpercentile. If this child later reaches the 5th percentile of weight,the expected pattern is absent. The ensuing diagnostic evaluation is aneffort to account for the deviation by finding a weight Pattern thatexplains all observations. These concepts extend easily to many othervalues, such as temperature. People have an average temperature of about37° C., but some are a little cooler and some a little warmer. Normaltemperature fluctuates within a narrow range during a lifetime, and mostdeviations from that range are considered abnormal.

Another example would be ST segments on a electrocardiogram. Followingan acute myocardial infarction, ST segments usually rise by varyingamounts, fall, and return to normal. The ST segment deviation from baseline varies with time and can be described by a Pattern, similar to thevariation in weights of growing children.

Many values change in predictable ways. Patterns might have cycles,sub-Patterns, and sub-sub-Patterns to describe these changes. Theaverage value of a variable often changes during a lifetime, while theinstantaneous value depends on a combination of annual, lunar, andcircadian cycles. For instance, a nonpregnant 20-year-old woman shouldexperience predictable lunar and circadian temperature fluctuations.

Sub-Patterns also describe consequences of other events, such as takinga drug. For instance, a dose of acetaminophen might lower a fever for 4hours. A fever responsive to acetaminophen could be modeled by ahigh-temperature Pattern with a sub-Pattern indicating 4 hours of normaltemperatures following acetaminophen doses. A person experiencing thisfever and taking acetaminophen every 4 hours maintains a normaltemperature. A physician observing this temperature Pattern would needto halt the acetaminophen to distinguish between a normal temperatureand fever responsive to acetaminophen.

Sub-Patterns characterize Features and therefore Findings. For instance,one of the chest pain Findings might be “crushing substernal chest painrelieved by rest or nitroglycerin and exacerbated by exertion.” Thisdescription implies a Finding with a designated location, a “crushing”Feature with some pattern, and 3 sub-Patterns describing the effect ofrest, nitroglycerin, and exercise. The clinical appearance of simulatedpatients with this Finding might still vary, depending on the allowedvariation in sub-patterns. For instance, pain might be more quicklyrelieved by nitroglycerin than rest or vice versa.

Finally, Patterns include Shape Selectors that help maintain consistencybetween variables. Shape Selectors are an example of Reasoning Elements,for example, small programs loosely based on the structure of Ardensyntax medical logical modules. See, for example, Johansson B G. WigertzO B, An Object oriented approach to interpret medical knowledge based onthe Arden syntax, Proc Annu Symp Comput Appl Med Car, 1992, pages 52-56,incorporated herein by reference.

Reasoning Elements define variables; assign their values from data aboutthe simulation; use loops, “if . . . then” statements, equations, andrandom numbers to reach conclusions; and finally produce some output. InFindings, the Shape Selector produces one percentile curve to representthe values of a Feature in an individual patient. For instance, althoughpediatric growth charts allow considerable variation in normal heightand weight, one child will exhibit a precise series of values for bothheight and weight. Height will closely track one percentile curve, aswill weight. The percentile of the height curve often limits thepossible percentiles of the weight curve: healthy children at 95thpercentile height rarely exhibit 5th percentile weight. Most childrenfollow a weight percentile equal to the height percentile _(—)20. Theweight Shape Selector can use this equation to restate the familiarheight-weight growth chart.

Patterns model time and are one approach to interrelated medicalobservations. Time affects most numeric values in the model.Consequently, Patterns appear in nearly every entity and relation.Patterns describe the incidence of diseases at different ages, thelikelihood of diseases progressing with time, and concentrations ofdrugs.

Courses of Action (COA) represent people's activities. Not only canthese activities be medical, such as taking a blood pressure orperforming a coronary artery bypass graft, but they can also includeattending school, working, asking and answering questions, and followingadvice.

Populations invoke Courses of Action to decide when to visit aphysician, how to answer questions, and whether to follow advice.Therefore, Courses of Action may advantageously be written to includemissed appointments, lying to physicians, and ignoring physician advice.These actions could even depend on aspects of the physician's conduct,such as how the physician chooses to obtain information.

Courses of Action have complex internal structures. A Course of Actionorganizes Step, which gather, process, and modify information aboutPopulations or Records. For example, a Step might be to obtain a bloodpressure from a person. Each Step uses a Reasoning Element to accomplishits tasks. In the case of obtaining a blood pressure, the ReasoningElement would determine and report the simulated patient's systolic anddiastolic blood pressure.

A group of Steps that can occur in any sequence is called a Batch. Forexample, when checking both right and left arm blood pressures, theorder in which the arms are checked is probably unimportant, so thesecan be distinct Steps within a Batch. The Course of Action lists aseries of Batches that must be executed in sequence, and describes anymandatory delays between Batches.

For example, to check orthostatic blood pressures, recumbent pressureswould be obtained in one Batch. The patient would sit or stand in asecond Batch. After a short mandatory delay, sitting or standingpressures would be obtained in a third Batch. Courses of Action alsodescribe possible earnings, costs, pleasure, and discomfort thatmotivate people to seek or avoid activities.

Agents include physical, chemical, biological, behavioral, and socialevents capable of influencing health States or Findings. These Agentscan be therapeutic, injurious, or both. Agent descriptions include dataabout intake, metabolism, and excretion, as applicable. For instance, along-acting steroid is a chemical agent. Following intramuscularinjection, the steroid will have predictable local and systemicconcentration Patterns as the chemical dissipates from the injectionsite. The steroid might be metabolized to other compounds and excreted.Exposure to Agents normally occurs during a Course of Action, as thisexample illustrates.

The model of Agents describes their recognition, their presence, and thepresence of metabolites or byproducts. Other parts of the model, such asthe sub-Patterns of Findings, describe the effects of Agents.

Table 1 lists relations shown in FIG. 3. The Health States Lead toHealth States relation describes how diseases evolve, and is therefore,critical for simulations. Preventive medicine scenarios might use thisrelation to generate patients who would benefit from screening. Casemanagement problems can use this relation to model both the past andevolving history of a patient. TABLE 1 Relations Between EntitiesPopulation Contacts Population Population Related to PopulationPopulation Interacts with Courses of Action Population Exposed to Agentsof Change Population Has Health States Population Exhibits FindingsAgents of Change Cause Health States Health States Lead to Health StatesFindings Associated with Health States Findings Link to Findings Courseof Action use Agents of Change Courses of Action Identify Agents ofChange Courses of Action Treat Health States Courses of Action AlterFindings Courses of Action Reveal Findings Courses of Action EvaluateFindingsNote: These relations link entities in the model together.

Unlike traditional knowledge bases, this relation links Findings (withtheir Patterns) to a Health State, rather than linking a range ofFinding values to a Health State. Sensitivity and specificity arerepresented as age dependent Patterns, rather than constants. Thesensitivity of a Finding will be lower and the specificity higher inthis model than in traditional knowledge bases.

The Findings Link to Findings relation describes causal associationsbetween Finding Patterns, such as “severe cough causes abdominal musclepain.” This relation contains data about causality, mechanisms, andtemporal constraints. This relation facilitates reasoning aboutFindings.

The Courses of Action Treat Health States relation illustrates means ofcuring Health States or preventing their progression. Treatmentstherefore modify probabilities in a lead to relation.

Courses of Action have three relations with Findings. The first, Alter,implies changing a Feature Pattern by invoking a sub-Pattern. Forexample, giving acetaminophen could alter a fever. The second relation,Reveal, links examining Courses of Action to the Findings they produce.For instance, a procedure called “taking a blood pressure” revealssystolic blood pressure. The third relation, Evaluate, links a Findingto a Course of Action that might be used to investigate its cause. Thisrelation would link a Finding of systolic hypertension to a Course ofAction describing its work-up.

The Population Contacts Population relation traces transmission ofcommunicable Agents and potentially beliefs. Population Is Related toPopulation describes biological and social relations and the history ofthose relations, and traces transmission of genetic Agents. These tworelations allow descriptions of arbitrarily defined families, witharbitrarily harmonious interactions.

The Population Interacts with Courses of Action relation describes whythe Population began the Course of Action, what the Courses of Actioncost interested parties, and how comfortable the Population was duringthe Courses of Action. This model allows a patient to remember anunpleasant experience and resist having it repeated. Because Courses ofAction can include negative (buying a therapy) or positive (receiving apaycheck) change in wealth, this relation is also capable of being usedto model patients' economic inability to follow medical advice.

The Population Exposed to Agents of Change relation describesperceptions about the exposure, knowledge of exposure, and the course ofAction responsible for the exposure. This relation can describe exactlyhow an Agent was distributed in, metabolized by, and excreted from thisPopulation.

The Population Has Health States relation includes the preceding HealthState, a list of Findings attributable to the Health State, and the ageat onset, diagnosis, and evolution of the Health State. Health Statesaffect different individuals in different ways, and treatment oftendepends on the patient's impairments and perceptions. Consequently, apatient's beliefs about disease progression and perceptions of a HealthState belong in the Has relation.

The Population Exhibits Findings relation has similar perceptionattributes. Perceptions can be divided into Dysutility and concern.Dysutility indicates a trade-off a patient would accept to return tonormal. Concern indicates a trade-off a patient would accept for fullreassurance that a Finding or Health State does not portend futureDysutility. For instance, a patient with a minor left-sided chest painmight rate its current Dysutility as $5 (“I would spend $5 to relievethis pain for today.”), and the concern as $100 (“I would spend $100 forassurance that nothing serious caused this pain.”). If the pain persistsunchanged, both of these values might decline as the patient learns tocope with the discomfort and becomes confident that the symptom has noprognostic importance. Thus, patients can have changing attitudes aboutstable conditions. Patients would typically seek medical care whenprovoked to so by a Dysutility or concern.

Records have the same relations as Populations, except that the detailsare always more ambiguous, inaccurate, or both. For instance, a patientmight have influenza starting December 15, while his Record of himselfindicates that he developed influenza between December 10 and December13. The patient's Record of himself is both ambiguous (there are 4possible days of onset) and incorrect (none of the days is December 15).

We have further determined that the data described in the Lead to,Associated with, and Link to, relations often change with medicalinterventions or other events. Modifiers describe events that cause apermanent variation in the expected history of these relations. Forinstance, an event might make evolution to another Health State more orless likely (regular low-dose aspirin reduces the risk of acutemyocardial infarction), or could permanently alter the likelihood ofexhibiting a finding (cardiac transplant prohibits myocardial ischemicpain). The dashed lines in FIG. 3 show Modifiers. The following examplesillustrate some modifiers (for example, Bayesian network from a Lead torelation, Bayesian network describing risk factors for progression, andthe like).

Population Interacts with Courses of Action modifies Health States Leadto Health States. An appendectomy alters the progression of acuteappendicitis to appendiceal rupture. For example, life-span-alteringinterventions always modify a Lead to relation.

Population Exhibits Findings can modify Health States Lead to HealthStates. For example, being overweight increases chances of developing adeep vein thrombosis or pulmonary embolism.

Population Has Health States can modify Health States Lead to HealthStates. Diabetes accelerates the onset of cardiovascular disease.

Population Has Health States can modify Findings Associated with HealthStates. Diabetic neuropathies diminish pain associated with myocardialinfarction or extremity injuries.

Modifications of these relations account for many benefits ascribed toreceiving medical care. Other benefits can occur when medicalinterventions temporarily decrease the severity o Findings.

The model described herein is intended to be a highly structured andrealistic representation of family medicine that will guide the designof the family practice knowledge base and support the generation andevaluation of recertification examinations. In this model, the followingare strong assumptions:

(1) Health States are discrete and distinguishable on the basis ofassociated Findings, which are also discrete and distinguishable on thebasis of the Patterns of their Features. (2) After choosing a percentilecurve in a Pattern to represent some value, the percentile does notchange substantially. (3) Changes in Patterns (e.g., the probability ofone Health State evolving to another) can be described for importantcombinations of risk factors, interventions, and time of occurrence. (4)Transitions from one Pattern to another can be estimated by simplemeans. (5) Modifying relations do not have important interactions withone another. (6) Highly developed anatomic and physiologic models arenot necessary, because associations between Findings provide the sameinformation.

Although the model should have clear places to store nearly allinteresting facts about family practice, test generation does notrequire a comprehensive description of all facts used in familypractice. The proposed test generates plausible problems from a set ofdata intentionally skewed to generate interesting (i.e., discriminating)cases. The present invention provides the flexibility to avoidcontroversial questions by controlling skewed data. For instance, if themanagement of borderline diabetes is controversial, the presentinvention allows editing of the family practice knowledge base so thatdiabetics' fasting blood glucose levels are always markedly elevated.The family practice knowledge base would then be incapable of creating aborderline diabetic.

The diagram of the model illustrated in FIG. 3 reflects many familymedicine concepts, and therefore, helps students, physicians and othersunderstand the process at work in family medicine. For instance, thediagram illustrates that Populations have biological and socialrelations. Populations exist in Health States, which evolve into new,sometimes undesired Health States.

A major goal of family medicine is to retard or stop undesirableevolutions and promote desirable evolutions. Stopping one undesirableevolution could, however, result in a different undesirable evolution.In addition, physicians who treat symptoms will Alter Findings, but donot necessarily Treat Health States. Altering Findings usually changescurrent quality of life, whereas treating Health States usually changesfuture quality and quantity of life. Because Findings occur in thecontext of Health States, we have determined that physicians contemplatewhat Health States might be responsible for Findings, rather than Alterthe Finding without considering future quality of life. The only toolsavailable for these causes are Courses of Action. Physicians prescribeCourses of Action, but only patients Interact with Courses of Action.For example, the prescription does not guarantee that the patientfollows the correct Course of Action. Agents (e.g., drugs) make adifference only when used in the context of a Course of Action.

The model's details provide further insights for students. First, timeis an extremely important element of primary care. Patterns become moredistinctive as time passes, simplifying diagnosis. The total risk ofgoing from one Health State to another increases with time, increasingthe value of early interventions. Second, patients have variable andevolving attitudes about Health States, Findings, and Courses of Action.The goal of medicine might not be to adhere to an endorsed Course ofAction, but to optimize each patient's perception of his or her qualityof life. To reach this goal, physicians adjust Courses of Action toaccommodate individuals' attitudes. Third, the importance of time andattitude in optimizing the quality of a patient's lifetime suggests thatcontinuity of care might help some patients.

The scope of family practice and the importance of protocols, time,individual variations and attitudes, and rationales distinguishes thecontent of the family practice knowledge base. That is, advantageously,some differential diagnosis of internally generated cases is possibleusing the model.

In this model, differential diagnosis largely depends on establishingthe presence of Findings, which in turn depends on establishing thepresence of Patterns and sub-Patterns of Features. Except in rare casesof pathognomonic values, confidence in the presence of a Pattern willincrease with the passage of time.

We have also determined that the structure of an interface to medicalreference systems might be enhanced using the model. Current referencesystems use the structure of medical publications and lists ofabstracted subject headings to facilitate searches through very largedatabases. These searches can yield large numbers of extraneouscitations, especially for novice users.

The model suggests an alternative indexing strategy, as well as agraphical search interface. For instance, one could view a queryinterface similar to FIG. 3. To request a query about the effect ofinsulin treatment on the development of retinopathy in diabeticpatients, one selects diabetes from an unrestricted list of HealthStates. The Lead to allows the user to select diabetic retinopathy froma list of diseases restricted to diabetic sequelae. The Modifierspecifies which Course of Action or Agent of Change to consider. Thecomputer delivers a list of references mentioning insulin in a diabetesLeads to diabetic retinopathy relation. Searching for a particularrelation between two entities improves the efficiency of searchesusually performed by naming the entities.

Overview of Patient Generation/Evolution Processes

We describe here an overview of processes used in thecertification/recertification system. The processes are divided intofour main groups:

-   1. Patient generation processes:    -   history outline processes    -   history generation processes-   2. Simulation processes    -   Presentation interface processes    -   Patient evolution processes

Patient generation processes are called once and produce the subject forthe examination session. Simulation processes may be called repeatedlyseveral times. The patient generation process presents the patient tothe examinee, collect the examinee's responses and queries, and evolvethe patient. See FIG. 4 for a pictorial overview of the system.

For the patient generation process, we assume that the area for thesimulation—a specific object, say A, from the class AREA—and a healthstate, say H, from the primary network of the area A are given. Forexample, A may be the area of the adult onset diabetes and H may be thehealth state of symptomatic diabetes.

The patient generation process consists of two phases:

-   -   1. history outline, and    -   2. history generation.

The goal of the history outline phase is to generate a progression ofhealth states and risk factors traversed by the patient on the way fromthe normal condition to the specified health state H. It starts with acall to the procedure that establishes sex and race of the patient beinggenerated (referred to as procedure GenderRace). The next stepestablishes the age of onset of H (call to procedure OnsetAge).

The goal of the next step is to select the precursor state for thetarget state in the simulation as well as risk factors (circumstances)that will affect the patient under construction. This will beaccomplished by a call to the procedure OutlineFirstStep.

The next procedure, OutlineGeneralStep, is called iteratively until thenormal health state is reached. In each iteration, it finds theprecursor health state as well as its onset time. When the normal healthstate is reached, the history outline phase is complete. See FIG. 5 fora flowchart of this process.

The GenderRace procedure generates sex and race of the patient underconstruction.

CreatePerson creates a basic description of the person. We select last,first and middle names, and age of the person, as well as two basicdemographic findings: sex and race. These last data are stored asEXHIBITS tuples (since demographic findings are treated as findings).

The OutlineFirstStep procedure generates the precursor state for thetarget health state for the simulation, and its onset age. In addition,it selects circumstances to which the simulated patient has beensubject. This procedure also creates an object HS_path, stored on thewhite board and containing the sequence of HAS instances for theprecursors of TS, starting with the normal health state and ending withTS. This sequence will be used later in the history generation phase.

The Generating history outline, and more specifically, theOutlineGeneralStep procedure, generates the complete path of precursorsof the target health state. It starts in the normal health state andterminates in the target health state TS (of course, the last but onestate on the path has already been generated by OutlineFirstStepprocedure).

History Generation

The history generation phase finds values that are established in eachcase when they differ from normal (normal values are derived from thedefaults maintained in the knowledge base). The general outline of thisphase is given in FIG. 6.

The reasoning element, called generation method, describing how a givenhealth state or a risk factor determines a finding, plays an importantrole in this phase. The generation method either provides a descriptionof all relevant basic features at all relevant sites (for normalstates), or determines which basic features at what sites need to beadjusted and by what specific findings. The main input for this phase isthe list of associated objects attached to the object P of type PERSON(the object of the simulation).

The history generation process looks at all associated objects andmodifies values of patterns describing relevant basic features so thatthe detailed description of the patient is consistent with the healthstate history as created in the earlier phase. Therefore, in this phasewe focus on describing findings and their basic features. To this end,we look at all health states represented by HAS instances. We sort themaccording to their onset times. This results in a list in which allstates normal in their areas precede all the abnormal states. The reasonfor this is that all normal states start at time 0. For each of thesenormal states we will run its generation method. This creates a list offinding names and site names to which the findings pertain, and definesthe domain of all findings for which specific descriptions are created.

Next, for every finding, the patterns of its basic features areinstantiated. We obtain these patterns from “normal” specific findingbelonging to the finding in question. To select specific curves, we usea percentile value. This value will generally be selected from, forexample, the range [0.15, 0.85] uniformly at random. Each time we needto use this value to select a specific pattern, we modify it, forexample, by a randomly selected number from the range [−0.05,0.05]. Inthis fashion the modified value is, for example, in the range[0.10,0.90].

After all normal states are processed, patterns of all basic features ofall relevant findings are instantiated for life. From now on, whenprocessing other health states these patterns are modified. The idea isto run the generation method for a health state. As a result we get alist of sites and basic features which must be modified as well asspecific findings where the new patterns can be found. If only somesites for the finding are generated, only those sites need to bemodified. To modify the patterns, we use patterns captured by theappropriate specific finding. Again the basic percentile is varied andused in the selection. The selected pattern is then superimposed on theexisting pattern (its values replace the old values starting with theonset age for the health state).

The generation method associated with the health state H, generates thelist of relevant findings with additional information on sites andspecific findings. That is, for each finding we maintain the list ofsites and with each of those we associate the list of all basic features(names) corresponding to the finding. Finally, these basic features aredescribed by their patterns.

The PatientDescription procedure selects HAS instances. It then arrangesthem according to onset times, generally earliest first. In thisprocess, the procedure invokes the generation method procedures for eachhealth state, thus creating EXHIBITS tuples describing findingsassociated with health states.

The InitPt Description (Initialize Patient Description) procedureinitializes the list PATIENT FINDINGS, which contains all findingsrelevant to the primary health state as well as all secondary(modifying) health states. It creates all corresponding EXHIBITSinstances and attaches them to the list associated_objects. All thesefindings are initialized to their normal values.

After the call to InitPtDescription, the domain of findings, sites andbasic features, which subsequently will be modified, is defined.CreatePtDescription scans the list of HAS instances and adjusts findingsso that the resulting patterns are consistent with the history of healthstates.

Patient Evolution

As explained earlier, we assume that data required by the processes isstored in the entity relationship model, white board (WB) and in thearea of memory local to patient generation and evolution processes. Thislocal memory will be denoted as LM. We start the evolution phase withthe patient fully described and stored in the WB. An equivalentdescription exists in LM. Several HAS instances describe continuinghealth states (one of them -primary). After the assessment phase(requiring physical examination and history taking) the examineeproposes treatment consisting of one or more courses of action. Thesecourses of action may alter some of the health states the patient iscurrently in. All selections made by the examinee are gathered in atable coa_list.

LEAD_TO data describes probabilistic information on progress from onehealth state to another. This data depends on modifiers. At present, weuse a small generic set of modifiers: “fast progress,” “moderateprogress” and “slow progress.” For each of these modifiers, and for anedge in the health state network between a precursor health state PS,and the target health state TS, the entity relationship model containsan estimate of the flow along that edge.

Courses of action are represented in WB by a table which describes theirstructure in terms of elementary courses of action. We will describethis structure below. In addition, each course of action contains areasoning element. This reasoning element, given an edge (a pair(PS,TS)) and a set of other current health states (as modifying events),computes one of these three modifiers. Flows on the edges starting inthe current health state are used in the selection process. Once theselection is made, duration risk stored in the appropriate LEAD_TO tupleis used to determine the onset time for the selected health state.

The following structure is used to represent a course of action COA inWB. The data is stored in a table with, for example, four columns(additional columns may be necessary later for evaluation purposes). Thefirst column is labeled ECOA (elementary course of action). It lists allconcrete elementary courses of action that might be used in aconstruction of COA. The second column describes the type of thecorresponding elementary course of action. ECOAs of the same type areidentified by the same integer in the second column. The third columncontains one of five boolean operators: none (NOR), single (XOR), atleast one (OR), some but not all (NAND), all (AND). All members of atype are assigned the same operator in column 3. The fourth columncontains weights which are used in the matching process.

One of the courses of action listed with every health state is calledTIME. It describes the effects of no specific action by the examinee andserves as a default course of action.

The evolution phase is accomplished by the procedure called Evolve.Evolve has three input parameters: patient P, patient's age T, and thelist coa_list of COAs selected by the examinee. Evolve starts bycreating the list of patient P continuing health states. This isaccomplished by the procedure called SelectPresentHas. SelectPresentHasselects from the list of P associated objects those HAS instances thatrepresent continuing health states. It arranges selected HAS instancesin a list.

For each health state PS described by the list of selected HASinstances, we then identify in all the courses of action that arerelevant to PS. It gathers all those courses of action that are inrelation MANAGE with the health state PS, in the list called, forexample, coas.

At this time, the closest COA, among those found relevant to PS, to theexaminee selection (described, recall, by the list coa_list) is chosen.For the course of action, say COA, target states are created for PS,corresponding modifiers and flows. This data is used for evolution.

These steps are repeated for each health state PS. When the process iscompleted, all successor health states are represented by means of thecorresponding HAS instances. The evolution step is completed with a callto CreateDescription procedure. It generates descriptions of specificfindings corresponding to the health states.

Stochastic Process for Patient History Generation

The present invention provides a method to automate authoring of majorevents in simulated medical histories. We have designed a knowledge basewith temporal descriptions of the incidence and prevalence of healthconditions and plausible intervals between health conditions. Eachhealth condition is part of a small sequence of related and mutuallyexclusive health conditions. Many of these small networks exist inparallel.

We have determined that a patient's overall health can be described by avector indicating the patient's current health condition in eachnetwork. A patient's location in one network often affects timing oftransitions in other networks. The knowledge base advantageously usesmodifiers (for example, Bayesian network from a Lead to relation,Bayesian network describing risk factors for progression, and the like)to describe the influence of these and other risk factors, as well asinterventions, on incidence and transition times. A stochastic historyoutlining algorithm uses these data to construct a lifetime and recentmedical history whereby a patient might develop a specified vector ofhealth conditions.

The present invention generates a large number of plausible historyoutlines. The present invention automates the authoring of major eventsin the lives of simulated patients. The present invention applies aMonte Carlo process to multiple stochastic trees, to generate largenumbers of plausible case outlines. Further automated embellishment ofthese outlines yields complete, usable simulated case histories.

Previous efforts to simulate patients from data have used sensitivityinformation stored in a diagnostic database, or Quick MedicalReference®, to stochastically create a description of findings in apatient with a disease. See, for example, Bergeron B. Iliad: ADiagnostic Consultant and Patient Simulator, MD Computing 1991, Vol. 8,pages 46-53; Miller R A, Masarie F E, Myers J D, “Quick MedicalReference (QMR)” for diagnostic assurance, MD Computing 1986, Vol. 5,pages 34-49, incorporated herein by reference. However, we havedetermined that these simulations lack rich historical details and maygenerate implausible combinations of events. See, for example, SumnerW., A review of Iliad and QMR for primary care providers, Archives ofFamily Medicine 1993, Vol. 2, pages 87-95, incorporated herein byreference.

Some simulations generate patient details from a complete and precisemathematical model of pathophysiology. See, for example, Valdivia T D,Hotchkiss J, Crooke P, Marini J., Simulating the clinical care ofpatients: A comprehensive mathematical model of human pathophysiology,Proc 19th Annu Symp Comput Appl Med Care. 1995, page 1015, incorporatedherein by reference. This elegant approach is feasible in intensivemedical care and some restricted organ systems, but primary careproblems are not so well understood at present, and therefore requireempirical description.

Accordingly, we have also developed a process for generating detailedpatient histories culminating in a specified set of simulated healthproblems. The first segment of the algorithm creates an outline of themedically important events in a patient's life, including the patient'sage at the onset and termination of different health conditions orexposures to biologically active agents. The second segment of thealgorithm yields a detailed description of continuously defined factsabout the patient, such as physical and chemical characteristics,morphology, function, and sensations throughout life.

The history outlining algorithm essentially creates paths throughtemporally reversed Monte-Carlo processes, casting major events in apatient's history while guaranteeing that the history ends withspecified medical conditions. See generally, Rubinstein R Y, Simulationand the Monte Carlo Method, New York, N.Y., John Wiley and Sons Inc.;1981, incorporated herein by reference. This process is applied to a setof stochastic disease history models, each describing the evolution ofone health problem.

A knowledge base stores these models, along with standard modifiers thatcalculate temporal constraints on disease progression, conditioned oncomorbidities and treatments. This algorithm is capable of generatingmany plausible cases in a short period of time preceding an examination.

The “Health condition Leads To Health condition” cycle is the centralcomponent in the generation of a patient history. A health condition isa named collection of facts, which usually have prognostic implications.Typically, the facts that connote a health condition have a specifieddegree of variation from normal ranges, and are thought to arise from acommon underlying cause. A health condition can usually be considered tobe located at one or more body structures where that underlying cause ispresent.

Health conditions uses patterns and subpatterns to predict theirprevalence and incidence, conditioned on factors such as sex and race.Prevalence and incidence are provided in a widely used structure calledshape, which plots a value over time. In this situation, time indicatesthe simulated patient's age.

A health condition uses a generation method reasoning element toestablish the facts pertaining to its instantiation. These facts mayinclude events like drinking alcohol or driving cars, but most facts arespecific instantiations of more generic medical concepts, such assymptoms or laboratory values, in specified body parts. For instance,the generic concept of “synovial fluid glucose level” might beinstantiated as “normal” in “both knees.” Shapes describe exactly how avalue in this instantiation may reasonably evolve or fluctuate overtime.

Two special classes of health conditions exist. First, normal healthconditions are incident only at birth (or conception, depending ontesting goals). Second, “Alive” is a health condition whose prevalenceshows the proportion of a cohort that survives to any age. The agespecific prevalence and incidence of all other health conditions aredefined as the percentage of living individuals at that age whoexperience or acquire the condition, respectively.

The leads to relation connects one health condition (the precursor) toanother (the target), and describes possible time intervals required forevolution from the precursor to the target. A Pattern describes aprobability density function (pdf) of these time intervals, conditionedon comorbidities, treatments, and other risk factors. This duration pdfprovides a time constraint mechanism. For instance, a duration pdf forthe progression of mild to moderate knee osteoarthritis, given obesity,might indicate a probability density of zero in the first five yearsfollowing the onset of mild osteoarthritis, a uniform probabilitydensity from year five to year twenty, and then a probability of zero.This implies that all simulated obese patients develop moderateosteoarthritis between five and twenty years after the onset of mildosteoarthritis, and forbids simulated onsets at other times.

The modifiers of a Lead to relation also provide time constraints forrisk factors. This allows the model to represent the concept thatobesity must exist for a period of at least 10 and up to 40 years forthis duration pdf to apply.

Finally, the Lead to relation provides information about how quickly andcompletely to convert from the findings typical of the precursor tofindings typical of the target. For instance, if each kneeosteoarthritis stage is a health condition, and each stage has a typicaldegree of joint space narrowing, then the transition from one stage toanother should be accompanied by more narrowing of the joint space. TheLead to relation can indicate that this narrowing occurs over years, andthat the narrowing is nearly complete when the simulation asserts thatthe latter osteoarthritis stage is present.

A series of Lead to relations connect health conditions into smallnetworks illustrating evolutionary sequences of events. These networksoften suggest a disease staging scheme, such as (Stage 0) No KneeOsteoarthritis, (Stage 1) Mild Knee Osteoarthritis, (Stage 2) ModerateKnee Osteoarthritis, and (Stage 3) Severe Knee Osteoarthritis.

We call this sequence a parallel health condition network. It is“parallel” to many other networks of health conditions that existsimultaneously in a person. In general, a parallel health conditionnetwork lists transitions that occur among an exhaustive set of mutuallyexclusive health conditions occurring in one body part. For instance,the left knee of a patient exists in one of the health conditions in theosteoarthritis network. The right knee also exists in one of theseconditions, but not necessarily the same condition found in the leftknee. The patient simultaneously exists with one condition in a gastriculcer network, a weight network, and numerous other networks.

A simulated patient's overall medical condition is therefore a vector,V, listing the current health condition from each parallel network ateach involved site. A case specifies vector V₀, indicating the healthconditions instantiated at the initial presentation of a simulatedpatient, and sufficient information to create a history of vectorsculminating in V₀.

Most of the parallel networks in any given case are inactive. Thesedefine an initial, usually normal, (stage 0) condition of the parallelnetwork. Most cases contain a few active parallel networks. Activenetworks presenting at stage 1 or higher represent active medicalproblems. Active networks presenting at stage 0 represent potentialproblems, such as complications resulting from an active problem or itstreatment. The examinee's task is generally to identify and respond toactive networks in advanced stages, while minimizing disease progressionin active networks at stage 0.

Active networks can be divided into two categories. A case usuallyfocuses on care for a primary network “P” (for instance, osteoarthritisof the knees). A comorbid network “C” usually includes health conditionsthat influence, or are influenced by, the stage of evolution of aprimary network. For instance, obesity is a risk factor forosteoarthritis, and osteoarthritis may worsen obesity by limitingexercise. Comorbid networks that do not interact with the primarynetwork in any important manner may serve as distractors.

For instance, an episode of urethritis might be irrelevant toosteoarthritis, but suggests Reiter's syndrome as an alternativeexplanation for knee pain with an effusion. An active, stage 0 comorbidnetwork provides opportunities for complications. For instance, asimulated osteoarthritis patient presenting with a “No gastric ulcer”health condition could advance to “Gastric ulcer” after receivingsteroidal nonsteroidal anti-inflammatory drugs.

When an active parallel network describes a chronic condition, acuteexacerbations may be expected with some of the health conditions in thenetwork. An exacerbation network “E” is a parallel network describingacute flares of illness that occur during a more chronic healthcondition. For instance, flares of knee pain with effusions may occur inpatients with chronic osteoarthritis. In principle, health conditionswithin an exacerbation network can have their own exacerbations. Thesimulation process of the present invention allows exacerbation networksto contain cycles, unlike primary and comorbid networks.

A simulated patient's medical history is the sum of the eventsculminating in the case defining vector, V₀. The case providessufficient information to create many plausible histories, but does notstore histories per se. Consider a case defined to culminate in severebilateral knee osteoarthritis and morbid obesity. The relative sequenceof events on the primary and comorbid networks are not necessarilyconstrained. Obesity might be required to occur before the onset of mildosteoarthritis. However, the onset of morbid obesity could occur beforeor after the onset of moderate osteoarthritis.

The Cartesian product of two active, linear parallel health conditionnetworks, P and C, yields a two dimensional web of health conditioncombinations. This product re-establishes the complexity avoided by theparallel network simplification, and calls attention to interactionsbetween P and C. A vertex in this web is composed of the ith healthcondition in P and the jth health condition in C, and is represented bythe vector V₀=(P_(i),C_(j)). Evolution can be assumed to occur in onlyone dimension at a time. If evolution in both networks can occursimultaneously in life, one can be assumed to occur first, and the othera moment later for purposes of the model. That is, the set of vectorsV⁻¹={(P_(i-1), C_(j))}; (P_(i), C_(j−1))} are immediate precursors ofvector V₀, but (P_(i-1), C_(j−1)) is not. Similarly, the set of vectorsV⁻² includes (P_(i-2), C_(j)), (P_(i-1), C_(j−1)) and (P_(i), C_(j−2)).

Three kinds of interaction are possible in the web formed by networks Pand C. First, the networks may be completely independent, so thatevolution along one dimension has no implications for evolution in theother. Second, progression through one network may depend on theconcurrent condition of an independent network. For instance, theincidence of early osteoarthritis conditions is dependent on thepresence of obesity. Finally, mutually dependent networks create a webin which progression through each network depends on the concurrentcondition of the other network. For instance, a realistic simulation ofa severe osteoarthritis history might require modeling a “vicious cycle”where obesity accelerates osteoarthritis, which in turn acceleratesobesity.

The Cartesian product of N parallel health condition networks similarlyyields an n-dimensional web of health condition combinations, withpotentially complex interactions. Data acquisition for these webs is adaunting task, but might be simplified by (1) limiting the number ofdimensions, (2) ignoring improbable health condition combinations,particularly when describing vicious cycles, and (3) assumingindependence for some kinds of test cases even when dependence exists inreality.

Stochastic Process History Outlining Process

The goal is to produce patient care scenarios for recertifyingdiplomates to manage. The data described above allow automaticgeneration of such cases, starting from a case specification. The caseis composed of primary network P, and comorbid health condition networkC. Network P is composed of health conditions P₀, . . . P_(n) and “leadto” relations PL_(0->1), . . . PL_(n-1→n). Network C is composed ofhealth conditions C_(o), . . . C_(m) and “lead to” relations CL_(0→1), .. . CL_(m-1→m).

Chronic health condition P_(j) in network P has acute flares describedby parallel network E. Network E is composed of conditions E₀, . . .E_(q) and “lead to” relations EL_(0->1), EL_(1->0), . . . EL_(q-1->q),EL_(q->q-1). The normal condition of network E is E₀, and the networkmay cycle through E₀ up to X times.

The vector V_(o)=(P_(i), C_(j), E_(k)) summarizes the health conditionsrequired at the presentation of the case. Health conditions P_(i) andE_(k) may be incident or prevalent at presentation. Incident healthconditions would typically require both diagnosis and management, whileprevalent health conditions would often be known diagnoses, and requireonly management decision. Health condition C_(j) is usually prevalent.

The first step assigns the sex, race, and other genetically determinedfacts to the prospective patient. If P_(i) is an incident healthcondition in the simulation, the incidence pattern for health conditionP_(i), is conditioned on sex and race. Sex and race are assigned byobtaining the area under the incidence curves for male and femalepatients of each race. The simulator makes a weighted random selectionof the patient's sex on the basis of the results.

In the weighted random selection process, a series of positive values isnormalized to one by dividing each value in the series by the sum of theseries. The resulting series defines a probability distribution. Toselect an item according to this probability distribution, the intervalfrom zero to one is divided into consecutive subintervals of lengthsequal to the corresponding probability the series. A random number fromzero to one is generated from the uniform distribution. The interval towhich it belongs defines the selected item.

Because the incidence or prevalence of some illnesses, such as kneeosteoarthritis, can increase dramatically with age, some correction toapproximate the absolute number of cases occurring at each age may beuseful, depending on the goals of the simulation. To obtain absolutenumbers of incident or prevalent cases at each age in a cohort, theincidence or prevalence at each age is multiplied by the fraction of thecohort in that age interval. Formula 1 illustrates this calculation, andthe general procedure for multiplying two shapes.

Formula 1. Absolute prevalence of health conditions as a function ofage:Absolute prevalence(P _(i) , n)=prevalence(P _(i) , n)*prevalence(Alive,n)Where prevalence (health condition, n)=the prevalence of healthcondition at age n years.

Similarly, the joint absolute prevalence of P_(i) and C_(j) can becalculated by multiplying the absolute prevalence of P_(i) by theprevalence of C_(j) in each age interval. Although the prevalence ofeither or both health conditions may be explicitly conditioned on thepresence of the other, knowledge acquisition efforts are unlikely tocapture such dependencies. Calculating the joint prevalence reduces thechance of creating an unsolvable history, for instance by creating aprevalent case of P_(i) at an age where C_(j) does not exist, regardlessof the prevalence produced in knowledge acquisition. A weighted randomselection of an age of presentation can be made from the product of theage specific prevalence of all representing health conditions, and thespecial condition “Alive.”

Often, either P_(i) or E_(k) is an incident health condition, and theage of onset of the presenting health condition vector, V₀=(P_(i),C_(j), E_(k)), is determined by the preceding step. In addition, theimmediately preceding health condition vector, V⁻¹, must be (P_(i-1),C_(j), E_(k)) if P_(i) is incident, because any other vector would makeP_(i) prevalent rather than incident at age N. More commonly, E_(k) isincident and vector V⁻¹ must be (P_(i), C_(j), E_(k-1)). Alternatively,if V₀ consist only of P_(i) prevalent health conditions, then the age ofonset of V₀ is unknown. In general, health condition vectors contain amixture of conditions with known ending times (e.g., precursors ofincident conditions in V₀) and unknown ending times (e.g., prevalentconditions in V₀).

Assume that P_(i) is an incident health condition at age N. Theinteresting vector is therefore V⁻¹=(P_(i-1), C_(j), E_(k)), becausehealth condition P_(i-1) evolved to P_(i) at age N. One possibleprecursor of vector V⁻¹ is (P_(i-2), C_(j), E_(k)) which would evolve tovector V⁻¹ at the age of onset of health condition P_(i-1).

The age of onset of P_(i-1) is constrained in part by the age specificincidence of P_(i-1), and N. The incidence of health condition P_(i-1),conditioned on race and sex yields the number of new cases per year pernumber of persons at risk, in each year from birth to age N. Because thesimulated patient must belong to a cohort of individuals who lived untilage N, corrections to obtain an absolute incidence are usually notimportant.

The age of onset of health condition P_(i-1) is further constrained bythe plausible duration of P_(i-1). For instance, if P_(i-1) alwaysprogresses to P_(i) within ten years, then a case of P_(i-1) must havebegun between ages (N−10) and N. The “lead to” relation PL_(i-1>i)provides a duration pdf, conditioned on pertinent facts representingsome known modifier. The duration pdf is a probability distributionfunction defining probabilities of evolution to P_(i) time intervalssubsequent to the development of P_(i-1). The duration pdf is truncatedat the time equivalent to the age of presentation, N (assuming thatP_(i) could not have begun before birth), and reversed in time. Thereversed duration pdf indicates at age 0 the probability that atransition from P_(i-1) to P_(i) would take N years, the simulatedpatient's entire life. In the year before presentation, at age N−1, thereversed duration pdf shows the probability that the transition wouldoccur after exactly one year.

For each year from birth to the age of onset of P_(i), the incidence ofhealth condition P_(i-1) and the reversed duration pdf are multiplied toobtain a weighting factor for the onset of P_(i-1) in that year. Theseweights are used to make a random weighted selection of one year topropose as the age of onset for the health condition P_(i-1). This agerepresents one proposal for the age of onset of V⁻¹=(P_(i), C_(j),E_(k)).

Formula 2. Weight (W_(n)) for establishing the onset of health conditionP_(i-1) at age n:W _(n)=Incidence(P _(i-1) , n)*DurationPDF (P _(i-1) , N−n)Where:

-   -   N=age of onset of health condition P_(i)    -   DurationPDF (health condition, x)=probability that health        condition evolves to its successor during the time interval x−1        to x years after its onset.

In general, this procedure is repeated for each health condition with anonset time after birth (or conception) in the currently interestingvector, V⁻¹. The result is a proposed list of ages of onset for a subsetof vectors in the set V⁻². The next step proposes ages of onset for theremaining vector in V⁻².

Assume that health condition C_(j) is a prevalent condition in asimulated patient presenting at age N. Assume that the annual incidenceof C_(j) is constant from age N−3 to N, and that C_(j) is equally likelyto evolve to C_(j+1) in 1, 2, or 3 years. The duration pdf from the“lead to” relation CL_(j->j+1) is therefore uniform over years 0 to 3.Consequently, C_(j) beginning at age N−3 is as likely to continue to ageN−2 as to age N−1, but will not be prevalent at age N in either case.Conversely, most cases of C_(j) beginning at age N−1 would be prevalentat age N. To accommodate the uncertainty regarding the onset time ofC_(j+1), the duration pdf is reversed in time (as in the previous step),then converted to a cumulative probability function. The highestcumulative probability occurs just before the age of presentation.

Formula 3. Reversed cumulative probability (RCP) of duration of healthcondition C_(j): RCP(n) = Σ(DurationPDF(CL_(j− > j + 1), N − y))y = 0  to  n

Where:

-   -   N=age at presentation    -   y=a number of years between 0 and n.

For each year from birth to the age of presentation, the incidence andreversed cumulative probability of duration are multiplied to obtain aweighting factor for the onset of C_(j) in that year, a random weightedselection chooses the year to propose as the age of onset for the healthcondition C_(j). This age represents a second proposal for the age ofonset of (P_(i)1, C_(j), E_(k)).

Formula 4. Weight (W_(n) for selecting age n for the onset of healthcondition C_(J):W _(n)=Incidence(C _(j) , n)*RCP(n)

At this point, the simulator has completed these steps. It found vectorV₀ to have a single possible predecessor, V⁻¹. Each health conditionlisted in V⁻¹ could have been the last to develop, therefore thesimulator proposed a plausible age of onset for each. The simulator usedone of two algorithms to calculate age of onset of each condition,depending on whether or not it could identify the age at which thecondition ended.

Each proposed age corresponds to a change in one element in vector V⁻¹.The collection of vectors produced by these single health conditionchanges is the set V⁻². Consequently, selecting the health condition tochange specifies which member of the set V⁻² is part o the history ofthis simulation. Although only one vector in V⁻² will appear in thehistory of this simulation, all of the health conditions in V⁻¹ will betraced back to birth through vectors from sets V⁻³, V⁻⁴, etc. Thequestion is not whether each condition has a history, but when eventsoccurred.

A safe strategy is to instantiate the vector from V⁻² occurring at thelatest age, along with any facts that had been tentatively proposed withthat age and vector. If two or more vectors from V⁻² share the latestmoment in age, one may be selected at random. The history generationstep is repeated with the instantiated vector from V⁻² replacing V⁻¹ asthe focus of attention.

The “lead to” relations, such as PL_(j-1->i), may need to instantiatemodifiers in order to produce a duration pdf.

Some modifiers might be defined by a history of a health condition in anactive network. Instantiations of health conditions in active networkscreate additional temporal constraints for these conditions. Theseconstraints typically dictate that a comorbid health condition, C_(x),is present at a point in time (e.g. at age N, the moment of transitionfrom P_(i-1) to P_(i)), for a period of time (e.g. at least five but notmore than ten years), or both (e.g. for the past two to four years).These conditions can be evaluated for logical compatibility withincidence data and the case. For instance, the instantiation of amodifier may require that C_(x) is present at the moment of transitionfrom P_(i-1) to P_(i). If x≠j and C_(j) is part of the target vector V₀,then this instantiation can not apply in this simulation. Theprobability of a modifier requiring C_(x·j) is therefore zero. Aslightly different constraint indicating that C_(x) is concurrent withP_(i-1) for five to ten years, where x=j−1, may be logically possible.

Note that the outlining algorithm will select this instantiation only ifthe onset of P_(i-1) is proposed for an older age than the onset ofC_(j). The simulator can therefore be required to add C_(j) at an olderage than the onset of P_(i-1). It is important to reconcile this age ofonset of C_(j) with incidence data for C_(j), before the tentativeinstantiation.

The simulation algorithm does not require that exacerbation networksreach any particular health condition prior to changes in their parentconditions. For instance, health condition P_(i) may permitexacerbations to reach condition E_(k), while health state P_(i-1) onlyallows exacerbations to reach condition E_(k-2). The simulationalgorithm may suggest that E_(k) developed before P_(i-1), creating anintermediate vector such as V_(i)={P_(i-1), . . . E_(k-1)}, which is inturn instantly preceded by V_(i2)={P_(i-1), . . . E_(k-2)}. Thesimulated medical history would indicate that the patient developedP_(i) and E_(k) simultaneously.

FIG. 7 is a flowchart providing an overview of the stochastic process.In FIG. 7, the stochastic process begins with defining a test area orsubject area to be tested in Step S2. In Step S4, the sex, race, andother genetically determined facts are assigned to the prospectivepatient. In Step S6, the past medical history of the patient isgenerated, by proposing concurrent histories for each of the healthconditions. In Step S8, the case history that will be accessible to theexaminee is generated for use in the examination.

In Step S10, the examinee or physician encounters the patient at apredetermined stage that is suitable for the examination. The examineemakes a decision as to whether treatment or intervention is appropriate,and either performs the treatment or not. The patient is optionallyevolved in Step S12 in accordance with the examinee's decision andactions performed in Step S10, and the examinee may be optionally testedagain in Step S10.

Stochastic Process History Outlining Example

Consider an examination of the management of osteoarthritis. Amongseveral cases in this area is one describing a patient with an acuteflare of osteoarthritis of the knee. The case presents with establishedgrade II chronic osteoarthritis, obesity, and No Gastric Ulcers. Noother networks are active in this case. The health conditions inparallel networks are:

-   P: Grade 0 Knee Osteoarthritis (OA), Grade I Knee OA, Grade II Knee    OA, Grade III Knee OA-   C: Normal weight, Obesity, Morbid Obesity-   C*: No Gastric Ulcer, Grade I gastric ulcer

The health conditions Grade I Knee OA and Grade II Knee OA areassociated with exacerbation networks:

-   -   ^(E)grade-II: Baseline Knee OA, Acute Flare of Knee OA    -   ^(E)grade-I: Baseline Knee OA

The presenting vector is $\begin{matrix}{V_{0} = \left\{ {P_{3},E_{2},C_{2},C_{1}^{\prime}} \right\}} \\{= \left\{ {{{Grade}\quad{II}\quad{Knee}\quad{OA}},{{Acute}\quad{Flare}\quad{of}\quad{Knee}\quad{OA}},{Obesity},} \right.} \\\left. {{No}\quad{Gastric}\quad{Ulcer}} \right\}\end{matrix}$

The “lead to” relations required for history generation are PL_(1->2),PL_(2->3), PL_(3->4); EL_(1->2), EL_(2->1); and CL_(1->2). The “lead to”relations required for evolution are PL_(3->4); EL_(2->1); CL_(2->1),CL_(2->3), and C*L_(1->2).

The normal health condition in the Egrade_(-II) exacerbation network,Baseline Knee OA, may be instantiated twice. The Acute Flare of Knee OAhealth condition is incident, and all other conditions are prevalent.

Age-specific prevalence data about the presenting health condition inthe primary network, Grade II Knee OA, conditioned on sex, race, andother essentially predetermined and generally permanent patientcharacteristics are provided.

The probability of generating a white female patient, given a case ofGrade II Knee OA is asserted to be 63%, the fraction of all OA casesfound to occur in white females.

When sex and race are selected, the state of the prevalence node isdefined. The prevalence node supplies the prevalence of Grade II Knee OAin white females as a shape defined by the points {(0 years, 0%); (25years, 0%); (35 years, 0.2%); (60 years, 5%); (100 years, 45%)}. Theprevalence of Grade II Knee OA at any specific age is found by linearinterpolation, so that the prevalence at age 20 is zero, and theprevalence at age 80 is 25%. The rapid rise in prevalence from age 60 to100 suggests a high probability of generating a very old patient,because these data do not reflect the scarcity of very old people.

To correctly simulate the age distribution of patients, an absoluteprevalence is calculated using formula 1. Assume that the prevalence ofthe special condition “Alive” for white females is a roughly sigmoidcurve with a median survival around 78 years, such as {(0 years, 100%);(1 week, 99.9%); (1 year, 99.8%); (15 years, 99.5%); 20 years, 99.2%);(50 years, 95%); (60 years, 85%); (80 years, 30%); (90 years, 8%); (99years, 05%); (100 years, 0%)}.

Formula 1 produces absolute prevalence weights including the points {(0years, 0); (2 years, 0%); (35 years, 0.2%); (50 years, 2.9%); (60 years,4.25%); (80 years, 7.5%); (90 years, 2.8%); (99 years, 0.2%); (100years, 0%)}. The peak absolute prevalence (8.77%) of Grade II Knee OAtherefore occurs at age 73 rather than age 100, and absolute prevalenceis skewed toward younger patients, so that the median age of prevalentcases is 71. The product of the Alive and Grade II Knee OA prevalence issimilarly multiplied by the prevalence of the Obesity and No GastricUlcer conditions. This could further skew the age distribution away fromthe elderly as obesity, a risk factor for death at relatively youngages, is less prevalent in older patients.

Finally, the incidence of Acute Flare of Knee OA is obtained, if it isavailable. Since this health condition is part of an exacerbationnetwork, it might be safely assumed to be equally likely to occur at anyage where its parent, Grade II Knee OA, is present, if the incidence ofE_(k) is not specified. In this case, no further adjustment to theprevalence product produced above is required.

In general, the incidence shape for an incident health condition can bemultiplied by the product of the prevalence shapes obtained above. Oneyear is chosen at random from the resulting distribution in a weightedrandom selection process. We will assume that the process selects age 70for this patient's presentation. This means that a white woman with ahistory of Grade II Knee OA, Obesity, and No Gastric Ulcer, presents atage 70 with an acute flare of her osteoarthritis.

The next process generates the past medical history of the patient, byproposing concurrent histories for each of the health conditions in thepresentation vector V₀={Grade II Knee OA, Acute Flare of Knee OA,Obesity, No Gastric Ulcer}. The first step in this process traces healthcondition transitions as illustrated in FIG. 8.

As illustrated in FIG. 8, the Acute Flare of Knee OA is incident, sothat its precursor, Baseline Knee OA, must be present in vectorV⁻¹={Grade II Knee OA, Baseline Knee OA, Obesity, No Gastric Ulcer}. Theage of onset of V₁ and the preceding vector V⁻² are obtainedsimultaneously by predicting when each element of V⁻¹ might havedeveloped, and asserting that the last predicted change did occur.

Grade II Knee OA, an element of vectors V₀ and V⁻¹, will eventuallyevolve to Grade III Knee OA. A history generating relation, Grade IIKnee OA leads to Grade III Knee OA, describes how long this might take,perhaps 5 to 10 years. If this relation posits a shorter intervalbetween these conditions, then the simulation is constrained to producepatients with a recent onset of Grade II Knee OA. If the historygenerating relation posits a longer interval, then patients may have along established osteoarthritis condition.

Grade II Knee OA is prevalent in vector V₀, presenting at age 70, andwith no more than 10 years allowed for evolution to Grade III knee OA,the earliest age at which the grade II condition could have appeared is60 years. If so, this patient remained a longer time than usual in GradeII Knee OA, and the transition to Grade III Knee OA is expected shortly.The patient is most likely to have developed Grade II Knee OA betweenage 65 and 70, among a cohort in which no one would have progressed toGrade III Knee OA by age 70. If the incidence of Grade II Knee OA risesfrom age 60 to 70, the product of the reversed cumulative PDF and theincidence shapes will be further skewed towards later ages. We willassume that age 65 years is randomly selected from this product.

A similar procedure produces an age of onset for obesity. A historygenerating relation, Obesity leads to Morbid Obesity, describes thelength of transitions, perhaps 10 to 25 years. Obesity is prevalent inV₀, so a reversed cumulative PDF is multiplied by the incidence ofObesity, and an onset age between 45 and 60 is proposed.

The No Gastric Ulcer element in V₀ is a stage 0 condition, which mightevolve to stage I at some time. Since the incidence of stage 0conditions is always between 0 and 100% at birth, but is always 0% afterbirth, so that the duration PDF is irrelevant to the selection of theage of onset, as long as the reversed cumulative duration PDF isnon-zero at birth.

Finally, the Acute Flare of Knee OA condition has a known onset time, atage 70. The history generating relation, Baseline Knee OA leads to AcuteFlare of Knee OA, describes the duration of Baseline Knee OA, perhaps 3to 12 months. If the duration of acute flares is very short, and thereare no other conditions in the exacerbation network, then this PDF alsodescribes the periodicity of flares, given the presence of Grade II KneeOA. If specific incidence data for the acute flare condition are notavailable, the incidence of the parent condition for the exacerbationnetwork (Grade II Knee OA) can be substituted. The product of thereversed (but not cumulative) duration PDF and the incidence supplies adistribution from which to select an age of onset, for instance 69years, 7 months. Since this is the oldest age proposed, it is selectedand instantiated. Step 2 of this process, illustrated in FIG. 9, isanalogous to Step 1 described above, and therefore, no additionaldiscussion is described herein.

Finding Generation for Stochastic Process

Finding generation adds detailed descriptions of patients' features tothe outline generated in the steps above. Beginning with a healthynewborn patient (or embryo) of the specified sex and race, the findinggeneration process assigns values of specific findings expected inhealthy individuals. These may change when the patient develops a newhealth condition at the age selected by the outlining process.

The patient's detailed features are generated using modelinginstructions stored as Reasoning elements with health conditions.Specific findings associated with normal health are created in asequence indicated by these instructions. Each Specific finding isinitially defined from the onset of life until age 100. For instance,the patient's height is derived from a randomly generated percentile anda set of shapes resembling a pediatric growth chart extended to age 100.The set of shapes used may be conditionally dependent on the sex, race,and any other established facts about the patient.

The finding generation process should generally create dependentfindings, e.g., knee pain, after generating the findings upon which theydepend, e.g., joint space narrowing. Careful selection of findings torepresent may reduce some dependencies. For example, the model ingeneral is more robust if height and body mass index are considered tobe independent findings, and weight is not calculated until explicitlyrequested during a simulation. Therefore, the model in general is morerobust if height and body mass index are considered to be independentfindings, and weight is not calculated until explicitly requested duringa simulation. Most findings are instantiated as a series of pairs ofvalues and ages. Values at other ages may be found by linearinterpolation.

Findings may vary with predictable circadian, lunar, and annual rhythms,described by shape subpatterns. Shape subpatterns can be combined with ashape to produce fluctuations on realistic temporal scales.

Finding distortions illustrate events having temporary effects on theshape of some value. For instance, a temperature shape during a febrileillness might be 39° C., with a distortion pattern indicating a 1° C.drop for four hours following administration of acetaminophen. The exacttemperature reported at a given time would depend on the current valueof the lifetime temperature shape and whether the patient consumedacetaminophen in the last four hours.

After determining patterns for all findings present at a point in time,the simulator proceeds forward in time to the next health conditionvector. The simulator updates findings for the new situation. This loopcontinues until the computer has described the findings of the patientin the final health condition vector.

Using Pre-Generated Patients

In accordance with one design of the present invention, when thecomputer based examination system generates and evolves a randompatient, it cannot reuse the patient information if the patient isevolved once. That is, every time the examination is executed, we needto generate a patient to continue the test. Not only does the process ofgenerating a patient take tremendous time, but also the evolved patientcannot generally be tested again in the future.

In accordance with another design of the invention, the patient ispre-generated, evolved and stored in the Whiteboard database. Thepresentation system can test the patient in countless time if wanted.Furthermore, different physicians can test the same patient at the sametime.

FIG. 10 is an illustration of the entity-relationship model whenpatients are not pre-generated, and FIG. 11 presents the modifiedentity-relation diagram of the modified Whiteboard database when thepatient is pre-generated. Each node represents a status of a patientwith parallel health states. For example, when a patient is generated,he or she is located at node 1, the patient might be evolved to severalstatus located at node 2, 3, 4 . . . , etc. Therefore, a patient canhave many nodes.

Many nodes can share same EXHIBITS and HAS. For instance, when a patientis evolved to a severe knee problem, we first take out the most updatedEXHIBITS of the previous node, modify it and then write it to the newnode, and at the same time generate a new EXHIBITS for the new node. Thenew node will point to the EXHIBITS prior to the most updated EXHIBITSof the previous node. If nodes are in the same content area, they alsoshare the same FINDINGS and PATTERNS, but their shapes are different,which can be found in table Pattern_Shape.

Since different physicians can use the same patient for the test at thesame time, the corresponding action contents needs to be given for eachphysician. Therefore, every time a patient has a new node, we alsogenerate the patient's action contents. When the physician gets to thepatient with the specific node, the action contents are copied tophysician_actions tables.

The table ACTIONS, HEALTHSTATE and ACTION_HEALTHSTATE are pre-generated,and a corresponding utility integrated with pre-generating COA iscreated. Accordingly, the evolution process for pre-generated patientsis, for example, as follows:

-   a) Based on the parallel health states of the patient at the    specific node, fetch all corresponding actionID from    action_healthstate.-   b) Based on the possible target of each actionID, construct all    combinations that lead to different parallel health states.-   c) Create a new node for each possible action combination.-   d) Copy the SHAPE from old node to the new node.-   e) Construct a tuple in table NodeToNode where the action    combination, old nodeID and new nodeID will be stored.    Generating Patients with Parallel Health State Networks

A detailed description of parallel health state networks is nowdescribed. We have determined that parallel health state networksprovide a model with a reasonable biological basis, more easily defineddata, greatly improved reuse potential, and a better segmentedimplementation. Evolution of synergistic health problems (e.g., viciouscycles) are managed using structures from the original data model. Aworking patient generation process is creatable using the parallelnetwork model.

We have determined that the number of conglomerate health states expandscombinatorially, and the incidence and duration of these conglomeratehealth states is often a matter of speculation or is redundant withpreviously stored information.

We have also determined that a parallel network approach improves on theaccessibility and reusability of health state data, while retaining theability to handle the dependencies inherent in synergistic cycles.

Humans are composed of inter-dependent cells organized into tissues andorgans. Some tissues directly or indirectly control the state of cellsin other organs through mechanical, neurohumoral, or other processes.

An individual's health reflects the current health of all of thesecells. Therefore, a very high resolution model of the life of a humanbody might describe the histories of the cells comprising the body,including their dependency on other cells. In clinically recognizableprocesses, the cells comprising one tissue share similar structure,function, and health with many of their immediate neighbors. Theirhealth may diverge rapidly from the health of the cells in othertissues. Therefore, a model concentrating on the histories of tissuesretains considerable resolution.

Each tissue can be imagined to evolve on its own standard scheduleunless some local insult occurs, or an insult to another tissue altersthe schedule. The normal tissue schedules proceed in parallel. Forinstance, bone, Islets of Langerhans, nephrons, and retinal tissue allgain and lose function at predetermined rates. If bone loses function(strength), a local pathological parallel process (fracture) becomesmore likely. If Islet cells lost function (insulin secretion), distantpathological parallel processes in nephrons and retinas become morelikely or progress more rapidly (diabetic nephropathy and retinopathy).

Without parallel networks, distractors, such as randomly appearing coldsor a history of appendicitis might require many conglomerate states.Also, information collected for one disease domain might have to becompletely replicated in other domains (for instance, obesitydescriptions would occur in osteoarthritis, diabetes, hypertension,combinations of the above, and independently).

We have also determined that many therapeutic complications are acutesite-specific illnesses superimposed on an antecedent illness. On theother hand, some problems interact in synergistic cycles: Osteoporosisincreases the likelihood of fractures, and immobility (following afracture) increases the rate of progression of osteoporosis.Consequently, many of the most interesting disease processes areintertwined with others. In a network of conglomerate health states,these dependencies can be explicitly described at nodes and along edgesbetween nodes. In a parallel network model, the interacting networksmust be aware of each other.

This view of health and function, we have determined, suggests adefinition of parallel health state networks: A parallel health statenetwork for a tissue describes a collection of clinically discoverableand mutually exclusive states in which that tissue may exist, andpossible transitions between states. For example, the normal developmentof a tissue, described from a person's birth to death, is one distinctstate in a network.

Physically separated cells of the same tissue type may exist in verydifferent states. For instance, the left and right knee joints aresusceptible to pathologically indistinguishable osteoarthritic changes,but one knee may exhibit more advanced changes than the other.Therefore, parallel networks require identification of involved sites.

A parallel network is, not coincidentally, a disease staging scheme.Parallel networks for chronic diseases are typically restatements offamiliar staging concepts (e.g., Stage 0 or no disease, followed byStage I or mild disease, and so on). The parallel network illustratesthese as sequential stages, even in acute processes such as anklesprains or burns. A third degree burn is always preceded by a seconddegree burn, if only for the briefest moment of time.

Parallel networks alter knowledge acquisition and storage requirements,as well as patient generation algorithms, when compared to conglomeratehealth state models. Diagnoses previously combined in a conglomeratestate become distinct states in different parallel networks. Theconglomerate health state of the body is described by a vectorindicating the current status of all parallel networks.

Illustrations of their disease domains help medical experts understandthe scope of their knowledge acquisition task. Initially intricatedomain models were decomposed into much less threatening parallelnetworks. FIG. 12 illustrates parallel network structures. The simplestnetwork is a collection of one or more static states, typical of genetic(e.g., Down's syndrome) and some congenital conditions (e.g.,anencephaly). The progressive network is a series of states with nocycles, typical of degenerative illnesses such as osteoarthritis. Thereversible network illustrates chronic but reversible conditions, suchas essential hypertension and weight disorders. In the injury network anacute insult evolves to either recovery or a chronic condition with alater recovery. Injury networks describe many infectious diseases andtrauma.

The addiction network illustrates that a person may abstain from, use,abuse, or become addicted to something; in the current model, apreviously addicted person can only be addicted or recovering, butcannot return to abstinence, use or abuse. The surgical interventionoverlay illustrates that new states can be added to the above networksusing irreversible therapies such as radiation or surgery.

Parallel networks of three types are identified. The primary networkcontains the diseases that define the domain, such as diabetes mellitus.The second type of network contains a risk factor for progressionthrough the primary network, such as obesity. The third type of networkincludes complications attributed to states in the primary network orits management, such as retinopathy.

We have also determined that the following information is used to createparallel networks: 1) how long a risk factor should exist before itcould influence a transition between states in a primary network, 2) thetime required for transitions in the primary network, given differentcombinations of risk factors, and 3) the number of passes an individualpatient should be allowed to make through a cycle (e.g., from acuteinjury to recovery back).

The data model objects originally intended to store risk factorsincluded a “Person HAS Health State” relation, which identified a healthstate, its onset and duration. In addition, HAS relations indicates apreceding HAS relation to support tracing of medical histories. Theseattributes are adapted to describe parallel synergistic networks.

The patient generation process uses a weighted random process to selectall times and events, starting with an age of onset for a health stateon the primary network. Risk factors are selected next. Unlike theconglomerate health state patient generation algorithm, any diagnosesassociated with altered risk must be described in a parallel network.The plausible range of duration for each risk factor is stored in a HASrelation, and used in selecting its onset age. If the risk factorevolves independently of the primary network, the HAS relation does notindicate a preceding HAS, and the algorithm creates the risk factorhistory using default assumptions in its parallel network. If theprimary network does interact with the risk factor, the preceding HASrelations provide time constraints that promote plausible concurrentevolution of the primary and risk factor networks.

The original history generation algorithms are used within independentlyevolving parallel networks. Consequently, the system continues tosupport conglomerate health states described as a parallel network. Incontrast to the conglomerate health state model, the parallel networktechnique may require explicit and separate generation of the historiesof the primary network and any number of risk factors.

Computer Implemented Process

The process of the computer based examination or assessment system isdescribed in detail in connection with FIGS. 13-14. The computerimplemented process includes the overall concept that the physician ispresented with an examination, and the process generates multipleinstances of patients. These generated patients represent clinicalscenarios that a physician would have to go through to administer propertreatment. These scenarios are stored in a white board database whichstores both the database implementation (i.e. the patients stored indata structures), as well as computer codes which operate from basestructures including information on physician.

There are three basic actors in the computer based examination system:physician, white board and patient generator. The physician/examineeinitiates the white board action by logging in. Once the examinee logsin, then the white board makes one or more requests to the patientgenerator. The white board generally provides the patient simulator withthe basic testing area. The patient generator then starts the process ofgenerating the patient and evolving backwards, and optionally forwardsin time for pre-generated patients. Thus, the computer based examinationsystem includes separate programming objects in the general C++programming sense for physician, the patient and the white board.

In general, the physician/examinee pre-registers to take theexamination, and provides (or the system already has stored) detailedbackground information on the physician, areas of weaknesses, priorexamination information, and the like. Thus, the physician logs-in tothe computer based examination system in Step P2, and the systemvalidates the physician in accordance with predetermined criteria, e.g.,user ID, password, correct examination, and the like.

The physician/examinee is either presented with an optional list(s) ofsubject areas for examination or mandatory subject areas for examinationin Step P4, responsive to information stored in the whiteboard databasevia requests thereto in Step P12. Alternatively, the examination areasmight be hidden, and the examinee might be told that this is a diabeticproblem, with certain management issues. The examinee may optionallyhave a series of selections, whether it is in terms of individualpatients or they could be in specific areas.

In some instances, the examinee may be provided a patient with somespecific statements about the patient. The computer implemented processmay optionally determine whether the physician has been examined before.If the answer is yes, then the physician might require, for example,five of fifteen specific subject areas for the examination, of which oneor more would be available for testing.

In addition, prior performance of the physician may also be consideredusing a pre-stored or generated physician profile via Step P6, andrequests to the prior physician performance via Step P8. The specificexam content is then requested in Step P10 responsive to at least one ofphysician profile, prior performance, content areas. Accordingly, one ormore of prior performance, the physician profile, the content of theexamination, are used to provide a selection list of the physician tochoose from in Step P14.

Depending on the above information, the patient generator process isthen initiated to create a patient for the examination in Step P16. Thepatient generation process may be performed in Step P18 in real-time foreach patient, or may be pre-generated as described above. Under thereal-time scenario, the selection of a problem area in Step P14translates into a target health state or area.

For example, if the problem area selected was diabetes, the targethealth state in the knowledge base would be diabetes. Using the targethealth state, there are generally a plurality of health statesassociated therewith. The computer implemented process then optionallyrandomly selects one of these areas as a precursor health state in StepP20. For example, a mild case of diabetes may be the precursor healthstate for normal health state of diabetes.

The selection of the precursor health state is based on, or calculates,onset age in Step P22 via incidence data in Step P24. The historygeneration computer process is a mechanism that sets up a reasonablebeginning time and ending time for the patient that is being presented.The computer process chooses a target health state, precursorinformation, sex and race from the target health state, and establishesthe age of the patient. The computer process then moves backwards intime to establish onset age when the condition occurs, and proceedsbackwards in time all the way to the normal state. Next, the processmoves forward in time to determine potential subsequent health statesfor the patient based on a variety of possible interventions performedby the examinee. Thus, the process has two stages.

Depending on the precursor information/health state, information such asthe sex and race, along with disease prevalence in Step P34, mortalitydata Step P36 and incidence information in Step P24, are used to selectthe specific sex and race for the simulation.

The mortality data is based on sex and race. The sex and race isselected from the health state, incidence or prevalence data and sex andrace specify mortality data. For example, if the health problem that ispresented to the examinee is new to the patient, then it is incident(e.g., a recently broken bone). Alternatively, if the health problem isan old established problem such as long term diabetes, the healthproblem is prevalent. Thus, the incidence and prevalence is insertedinto the patient case history over and over again depending on theparticular problem. Accordingly, a pre-determined decision is generallymade as to what types of disease are to be tested, prevalent disease orincident disease.

The sex and race selection uses the disease prevalence and mortalitydata in the sex and race selection process. The mortality data anddisease prevalence are used to establish the reasonable ages and sexes,and also ages of on[-]set. For example, this mechanism preventsgeneration of pregnant males, 14 year old Type-II diabetics, and thelike.

For example, the computer system determines, or is instructed asdescribed previously, that the problem area is arthritis. The sex, race,and age of the patient are determined, for example, at the point in timewhere treatment may be necessary. The patient history is then generatedback through the process/time to establish onset times of the variousdifferent health states. That is, from, for example, the point in timewhere the arthritis is severe, the patient history is generated at apoint when the arthritis was mild, and back to when the arthritis wassubstantially normal.

When the sex and race selection process is completed via the combinationof sex and race selection in Step P38 and onset age calculation in StepP40, a patient has been generated at a specific point in time with aspecific health state problem and the characteristics of that problem.Thus, the computer process has generated the patient, moved backwards intime from the disease onset age all the way back to normal. For example,if the computer process started with a mild condition for a specificdisease, the computer process goes backward one time interval to normalfrom mild. If the computer process begins with moderate, the computerprocess will move backward in time from moderate.

As a result of the computer process, a patient template is alsogenerated in Step P26 using the onset age determination in P40 and sexand race selection in Step P38. In addition, the generated patient isgiven a name in Step P28, and age including a date of birth in Step P30.The physician/examinee is then provided with the history of the patientfor use in diagnosing or prescribing treatment for the generatedpatient. The patient history includes, for example, age of the patient,race and sex. Up to this point in the computer process, the patient iscreated. From this point of the examination/computer process andforward, the patient and physician's interaction with that patientdetermine both the information provided to the physician/examinee, aswell as potential evolution of the patient. Changes in the patient'scharacteristics is a function of physician's action or inactions usingthe evolutionary process described below.

The evolutionary process is performed using the knowledge base structureor entity relationship model described in detail above. The knowledgebase structure has been separated from the white board structuresdescribed above for administrative purposes, but alternatively may alsobe combined therein. The knowledge base represents all the informationthat does not necessarily have to go with the patient for purposes ofpresentation to the examinee. The knowledge base includes informationused to create the patient and provide instances of information.

However, separating the knowledge base from the white board structurehas the advantage that the computer generated patients do not require asmuch data to be transported therewith. Accordingly, a separate structureis created called the white board structure. The white board structureadvantageously includes the information required to generate thepatients and to present the patients to the physician/examinee. Thewhite board structure includes information containing patientdescription and all the findings that are typically generated that arenot necessarily related to the problem, for example, blood pressure,blood glucose, and the like.

That is, the white board structure provides all information that isgenerally available to the examinee, such as information satisfyingexaminee queries on prior history, laboratory tests, and the like. Inaddition, when pre-generated patients are used, all findings associatedwith the patient including all pre-generated evolutionary states arealso stored in the white board data structure.

For example, if the patient had moderate arthritis, the patient maygenerally transition to two other health states: severe arthritis, ormild arthritis. Thus, in one embodiment of the invention, the computerprocess pre-generates the possible health states for the patient.According to this embodiment where the patient is pre-generated, theprocess of evolving a patient may, in some circumstances, be morecomputationally efficient than to generate the patients dynamically.Thus, for pre-generated patients described above in detail, all possiblestates are generated ahead of time and then used by the white boardstructure in accordance with the pre-generated state when activated orselected by the examinee.

The white board accesses the patient template in Step P42, and generatesthe patient record in Step P46, responsive to requests initiated by thewhite board to the patient history information in Step P44. The patientrecord is not generally reviewable by the examinee, except on individualrequests by the examinee in Step P48. The examinee requests informationfrom the patient record in Step 48 which provides the examinee thephysical view of the patient. For example, the patient's blood pressuremay be stored in the patient record for retrieval by the examinee. Otherexamples of information stored in the patient record include chiefcomplaint, past medical history, past patient behavior or complianceinformation.

The white board will also generate examinee actions and patientinterventions in Step P52 by reviewing and evaluating the physicianintervention in Step P50, responsive to the patient record. The examineeactions and patient interventions contribute to the patient evolutionconditions used in the patient evolution process described above indetail.

Whether the patient is pre-generated or not, the computerprocess/patient generator generates the initial patient, andsubsequently evolves the patient, and subsequently presents same to theexaminee. The patient is generated by the patient generator accessingthe patient evolution conditions in Step P54, the target health state inStep P56, and any existing parallel health states in Step P58. Thepatient is evolved by the patient generator in Step P60 to the evolvedhealth state, which may become the target health state in Step P62.

At this point we have a patient on the white board presented with aparticular health state, which typically is the form of a chiefcomplaint. From this time on, the examinee/physician takes control ofthe process, and nothing is going to happen in the computer basedexamination system unless the examinee/physician does something, unlessthe health state is time dependent and able to advance to another stateautomatically, such as by inaction on the part of the examinee.

For example, if the health state is an acute problem, such as a heartattack, there may be a time dependency built in that is going to forcesome action of the physician within a specific time before the patientexperiences another heart attack. In this example, the examinee usingthe computer based examination system may dismiss the patient, thepatient will walk out of the doctor's office/hospital, and the examineewould receive notification that the patient just showed up in theemergency room with a problem.

Alternatively, if the examinee is too slow in diagnosing an illness, theinability to treat the patient in a short period of time may also resultin the patient progressing to a different health state. For example, apatient that has a heart attack might progress to a more serious stateif the examinee does not perform corrective measures very quickly whilethe patient is, for example, in the hospital. In general, allowing timeto elapse without intervention is an intervention choice along with theother active interventions that an examinee might choose.

In order to determine the target health state, the “iterate until normalreached” process is initiated via Step P64 which sets one or morepre-cursor health states to the target health state. The “iterate untilnormal reached” process iterates in Step P66 until the normal healthstate is reached backward from the target health state. For example, ifa mild health state is selected, the precursor health state is normal.The “iterate until normal reached” process also establishes one or moreoptional parallel health states in Step P68. Precursor parallel healthstates are then generated as needed in Step P70, which are then used tocontribute to the patient history in Step P72.

The computer based examination system ensures that the age of onset forthe various parallel health states is reasonable. Thus, the process ofgenerating precursor health states for the parallel health state is amulti-dimensional process of monitoring health states to be consistent,to prevent unreasonable scenarios, time frames, and the like. If theparallel health states are related, they have to be related to eachother sufficiently enough so that the evolution of health states makessense.

The parallel health states are also used to establish the findings inStep P74, which contribute to the patient history in Step P76. While theabove steps have been described in, more or less, a sequential manner,it should be clear that the various steps described herein may beperformed in parallel, independently, and/or non-sequentially, as neededor for computational efficiency.

Advantageously, the computer implemented process includes the capabilityof utilizing parallel health states as part of the patient generationprocess, which is described above in detail. As part of the generationprocess, a decision is generally made to include or exclude thoseparticular parallel (e.g., morbid or co-morbid) health states along withthe original state of the disease.

We have determined that sometimes health problems tend to be concurrent,but they are not generally defined as being necessarily interrelated.The computer based examination system provides the feature of handling aplurality of health states, either related or not related to each other.For example, there tend to be lots of people with diabetes and highblood pressure. Accordingly, we define these two health states asrelated to each other. Alternatively, the plurality of health states maybe considered to be substantially independent and still within the scopeof the computer based examination system of the present invention.

The present invention further provides the feature of dealing withparallel health states substantially or completely independent of eachother to permit dependent or independent management decisions. Forexample, a person that has diabetes and high blood pressure generallyrequires slightly different management decisions than a person who justhas high blood pressure or a person who just has diabetes. For example,the physician/examinee may prescribe a more expensive anti-hypertensivedrug if the patient has both diabetes and high blood pressure because ofpotential complications unique to the combination of health states.Thus, the computer based examination system may be used to determinewhether the examinee has made the appropriate management decision.Alternatively, the computer based system may be used to collect variousresponses from different well recognized physicians to establish aminimum level of care for insurance companies, health careorganizations, other physicians, and the like.

The present invention also provides the feature of providingdistractions when attempting to diagnose the disease/illness. That is,the patient may include symptoms and/or indications that might berelated to the problem seemingly presented to the examinee, but, infact, these indications distract the examinee along an inappropriatepath, such as excessive testing, over-prescribing medications, and thelike. Accordingly, we have also determined that distraction makes a goodargument for having parallel problems.

At this point and time, the computer based examination system selects anarea for examination, and is in the process of working backwards intime. The process iterates from precursor health state down to thenormal health state, where at each precursor state the process considerspotential co-morbid problems. Both the precursor or subsequent healthstates are the primary problem, and the parallel health states generatefindings. The findings are a part of the patient history. For example, afinding of obesity might be a change in weight. The process movesbackwards while at the same time looking at potential parallel healthstates have been substantiated. The history of findings are generated atthe white board level.

Now if the physician takes some action at this point in time that causespatient evolution (that is, the physician causes some action which thewhite board is checking at this point and time), the white board matchesthe action up against something that is going to cause patientevolutionary health state change. The white board then makes a requestto the patient generator to evolve the health state.

If the full patient has been generated on the white board, then thepatient generator is replaced with the white board itself to provide apre-evolved patient from memory. If, however, the knowledge base islinked for a dynamic situation, then the patient generator dynamicallyevolves the patient. In either situation, the evolved health statebecomes the target health state at this time. For example, the healthstate has evolved from mild to moderate arthritis, or from mild obesityto moderate obesity.

The computer implemented process also includes the possibility oftreating patients with management health issues that do not generallybecome totally normal (e.g., long term diabetes, arthritis, and thelike), as well as health conditions that may return to completely normal(e.g., broken bone, and the like).

In fact, we have determined that it is particularly likely that patientswill revert to normal conditions when the patient experiences anexacerbation health state/condition for the computer based examinationsystem. For example, we have determined that an exacerbation conditioncan have, for example, mild, moderate and severe states. If the patienthas a moderate exacerbation, there is a chance that the patientexperienced a mild exacerbation before evolving to the moderate state.There is also a chance that the patient had a severe exacerbation, isnow recovering, and may return to the normal state a few minutes later.

In summary, the computer based examination system utilizes three actors,the white board the physician and the patient generator. The examineeinitiates the whole process by logging in. The examinee logs into thewhite board, and the white board accesses various information todetermine whether the examinee is valid. If the examinee is valid orverified, the white board looks up the examinee's profile to determineany background specifics on the examinee, such as specific areas needingimprovement, past examination results, and the like.

The white board then determines or is provided the exam content, andthen contacts the patient generator. The patient generator begins thegeneration process, selects the disease or subject area, and controlsthe actual combinations of health states and co-morbid health states viaa case structure. The case structure controls both the presenting healthstate as well as the co-morbid health state. The case structure filtersthe generation process and makes a predetermination to eliminatepredetermined impossible situations, or difficult or unimportantsituations that are not to be used in the testing. The case structureindicates that even though a specific health state or parallel healthstate is in the knowledge base and even potentially legitimate, the casestructure will not present that problem. Thus, the case structure simplycontrols which of the health states will be presented to the examinee,and which of the co-morbid health states, and possibly flare states willalso be presented to the examinee simultaneously or sequentially.

The white board then retrieves the patient template including, forexample, the patient history, the chief complaint, the assessment test,and the like. From this time on, the examinee performs some action byeither requesting data which is controlled by the white board or bycausing, directly or indirectly, some action to take place. Once anaction is performed, the patient may be evolved to the next health stateby the patient evolution process.

Both the request of information and the review and evaluation of theexaminee's actions or intervention are generally handled by the whiteboard for convenience, but multiple control mechanisms may also be used.If the white board sees there has been a change in health state for thepatient, then the white board would then go to patient evolution processto initiate the evolution, and request the patient generator to provideinformation regarding the evolved patient.

The patient evolution information may optionally be pre-generated forcomputational efficiency. That is, even when patients are createddynamically, some predetermined evolution information may be ready foruse by the computer based examination system based on the potential ofthe possible evolution periods/health states. For example, if the targethealth state was moderate, the computer based examination system mayhave predetermined onset time of moderate, and therefore, know the timefor mild, normal, severe, and the like. In effect, the white boarddatabase/object optionally includes a complete possible look at thefuture appearance, as well as the past characteristics for thisparticular patient at a particular health state.

The underlying goal of the computer based examination system is that theevolutionary process is generally the same as the patient generationprocess. Both processes are generally the same, just the generationprocess has more steps to generate the patient. In the evolutionarysituation, the computer based examination system deals with multiplepossible health state successors in different parallel networks. Thezero state, or state where the examination begins, generally has aprimary health state like moderate arthritis, possibly a flare statesuch as an acute swelling in the knee, and comorbid states such asoverweight.

To generate the patient history, the computer process take the moderatearthritis, the flared up knee, and the overweight condition and looksbackwards in time to determine the most recent precursor state. Forexample, the precursor of the moderate arthritis could be mild, theprecursors of the flare could be baseline or normal, and the precursorfor the overweight condition could be normal weight. The computerprocess sets the patient's current age, for example, as age 50, and nowmoves backward in time.

For example, for a 50 year old person with moderate arthritis, it islikely that the arthritis began 5-10 years ago. With respect to theflared knee, it is likely that this condition began within the lastcouple of days. For the overweight condition, it is likely that the 50year old person began this condition 10-15 years ago. Therefore, thereis a 5-10 year interval (arthritis), a 3 day interval (flared knee), anda 10-15 year interval (overweight). The computer process moves backwardin time to that last change that should have occurred. In thissituation, the first precursor health state is the flared knee whichoccurred 3 days ago. The clock then gets reset, and the next earliestprecursor health state is determined. The whiteboard generally throwsaway all the previous information that was used to generate the laterprecursor health state, and recalculates the next earliest precursorhealth state until all precursor health states are generated to thenormal condition for all conditions/diseases.

Switching to the forward version, the patient evolution process, thecomputer process looks forward in time instead of backwards in time.Therefore, considering the above example, there may be a change in 3days (knee flare), another change in 5-10 years (arthritis), and anotherchange in 10-15 years (overweight). The next change that will occur willbe in 3 days. That is the evolutionary process which, similar to thepatient history generation process, recalculates onset times for eachsubsequent health state. Thus, the main difference between the historygeneration process and the patient evolution process is the data beingapplied to each process.

The computer based examination system may also be used to determinewhether specific physicians are practicing cost effective medicine foruse by, for example, insurance companies. The system can provideobjective criteria for treating patients by defining episodes of carefor isolated problems. For example, the computer system can indicateapproximately the amount of money to spend on a patient with a heartattack with no other concurrent problems, for an asthmatic patient peryear, and the like. The computer based examination system and processprovides a “flight simulator” where the physician can practice specificpreferred forms of treatment, as appropriate. For example, if thepatient has a heart attack, the examinee/physician should generallyprescribe aspirin for long term usage, but many do not. Thus, thecomputer based examination system may also be used as a training systemso that the examinees rehearse a desirable behavior such as prescribingaspirin after heart attacks. The computer based examination system cantherefore also be used to increase desirable behavior when the physicianinteracts with a real patient.

Consequently, if a particular physician or group of physicians aredetermined to be expensive for an insurance company or health caregroup, and the computer based system shows that the physicians arelikely to provide appropriate care, then it is possible that thisphysician or group of physicians have a particularly expensive patientpopulation, and should therefore not be faulted. Further, the computerbased examination system may also be adapted to receive the specificdata collected by the physician and interventions associated therewith,to further verify that the practice is delivering the appropriateservices. Thus, the computer based examination system, may be used todetermine whether a particular practice is delivering services withinpredetermined guidelines.

FIG. 15 is an illustration of a main central processing unit forimplementing the computer processing in accordance with a computerimplemented embodiment of the present invention. The proceduresdescribed above may be presented in terms of program procedures executedon, for example, a computer or network of computers.

Viewed externally in FIG. 15, a computer system designated by referencenumeral 40 has a central processing unit 42 having disk drives 44 and46. Disk drive indications 44 and 46 are merely symbolic of a number ofdisk drives which might be accommodated by the computer system.Typically these would include a floppy disk drive such as 44, a harddisk drive (not shown externally) and a CD ROM indicated by slot 46. Thenumber and type of drives varies, typically with different computerconfigurations. Disk drives 44 and 46 are in fact optional, and forspace considerations, may easily be omitted from the computer systemused in conjunction with the process/apparatus described herein.

The computer also has an optional display 48 upon which information isdisplayed. In some situations, a keyboard 50 and a mouse 52 may beprovided as input devices to interface with the central processing unit42. Then again, for enhanced portability, the keyboard 50 may be eithera limited function keyboard or omitted in its entirety. In addition,mouse 52 may be a touch pad control device, or a track ball device, oreven omitted in its entirety as well. In addition, the computer systemalso optionally includes at least one infrared transmitter 76 and/orinfrared receiver 78 for either transmitting and/or receiving infraredsignals, as described below.

FIG. 16 illustrates a block diagram of the internal hardware of thecomputer of FIG. 15. A bus 56 serves as the main information highwayinterconnecting the other components of the computer. CPU 58 is thecentral processing unit of the system, performing calculations and logicoperations required to execute a program. Read only memory (ROM) 60 andrandom access memory (RAM) 62 constitute the main memory of thecomputer. Disk controller 64 interfaces one or more disk drives to thesystem bus 56. These disk drives may be floppy disk drives such as 70,or CD ROM or DVD (digital video disks) drive such as 66, or internal orexternal hard drives 68. As indicated previously, these various diskdrives and disk controllers are optional devices.

A display interface 72 interfaces display 48 and permits informationfrom the bus 56 to be displayed on the display 48. Again as indicated,display 48 is also an optional accessory. For example, display 48 couldbe substituted or omitted. Communication with external devices, forexample, the components of the apparatus described herein, occursutilizing communication port 74. For example, optical fibers and/orelectrical cables and/or conductors and/or optical communication (e.g.,infrared, and the like) and/or wireless communication (e.g., radiofrequency (RF), and the like) can be used as the transport mediumbetween the external devices and communication port 74.

In addition to the standard components of the computer, the computeralso optionally includes at least one of infrared transmitter 76 orinfrared receiver 78. Infrared transmitter 76 is utilized when thecomputer system is used in conjunction with one or more of theprocessing components/stations that transmits/receives data via infraredsignal transmission.

FIG. 17 is a block diagram of the internal hardware of the computer ofFIG. 15 in accordance with a second embodiment. In FIG. 17, instead ofutilizing an infrared transmitter or infrared receiver, the computersystem uses at least one of a low power radio transmitter 80 and/or alow power radio receiver 82. The low power radio transmitter 80transmits the signal for reception by components of the process, andreceives signals from the components via the low power radio receiver82. The low power radio transmitter and/or receiver 80, 82 are standarddevices in industry.

FIG. 18 is an illustration of an exemplary memory medium which can beused with disk drives illustrated in FIGS. 15-17. Typically, memorymedia such as floppy disks, or a CD ROM, or a digital video disk willcontain, for example, a multi-byte locale for a single byte language andthe program information for controlling the computer to enable thecomputer to perform the functions described herein. Alternatively, ROM60 and/or RAM 62′ illustrated in FIGS. 16-17 can also be used to storethe program information that is used to instruct the central processingunit 58 to perform the operations associated with the process.

Although processing system 40 is illustrated having a single processor,a single hard disk drive and a single local memory, processing system 40may suitably be equipped with any multitude or combination of processorsor storage devices. Processing system 40 may, in point of fact, bereplaced by, or combined with, any suitable processing system operativein accordance with the principles of the present invention, includingsophisticated calculators, and hand-held, laptop/notebook, mini,mainframe and super computers, as well as processing system networkcombinations of the same.

Conventional processing system architecture is more fully discussed inComputer Organization and Architecture, by William Stallings, MacMillamPublishing Co. (3rd ed. 1993); conventional processing system networkdesign is more fully discussed in Data Network Design, by Darren L.Spohn, McGraw-Hill, Inc. (1993), and conventional data communications ismore fully discussed in Data Communications Principles, by R. D. Gitlin,J. F. Hayes and S. B. Weinstain, Plenum Press (1992) and in The IrwinHandbook of Telecommunications, by James Harry Green, Irwin ProfessionalPublishing (2nd ed. 1992). Each of the foregoing publications isincorporated herein by reference.

Alternatively, the hardware configuration may be arranged according tothe multiple instruction multiple data (MIMD) multiprocessor format foradditional computing efficiency. The details of this form of computerarchitecture are disclosed in greater detail in, for example, U.S. Pat.No. 5,163,131; Boxer, A., Where Buses Cannot Go, IEEE Spectrum, February1995, pp. 41-45; and Barroso, L. A. et al., RPM: A Rapid PrototypingEngine for Multiprocessor Systems, IEEE Computer February 1995, pp.26-34, all of which are incorporated herein by reference.

In alternate preferred embodiments, the above-identified processor, andin particular microprocessing circuit 58, may be replaced by or combinedwith any other suitable processing circuits, including programmablelogic devices, such as PALs (programmable array logic) and PLAs(programmable logic arrays). DSPs (digital signal processors), FPGAs(field programmable gate arrays), ASICs (application specific integratedcircuits), VLSIs (very large scale integrated circuits) or the like.

The many features and advantages of the invention are apparent from thedetailed specification, and thus, it is intended by the appended claimsto cover all such features and advantages of the invention which fallwithin the true spirit and scope of the invention. Further, sincenumerous modifications and variations will readily occur to thoseskilled in the art, it is not desired to limit the invention to theexact construction and operation illustrated and described, andaccordingly, all suitable modifications and equivalents may be resortedto, falling within the scope of the invention.

For example, while the above discussion has separated the variousfunctions into separate functionality, the functions may be combined,physically and/or logically, and various functions may be combinedtogether. While combining various functions may make implementationdetails more cumbersome, nevertheless, the functions described hereinmay still be accomplished to advantageously provide some or all of thebenefits of the invention described herein.

As an additional example, the foregoing discussion focused exclusivelyon medical applications of the current invention. Advantageously, theinvention applies equally well to creating simulations of other complexsystems, particularly complex systems in which an empiric description iseasier to obtain than a comprehensive mathematical description. Theconcepts in the invention correspond to generic concepts that apply tocomplex systems in general. The labels in the current invention and thegeneric concept are listed in the table below.

The Population (or Person or Simulated Patient) concept represents anycomplex system. Consider a nuclear power plant. All breeder reactorsform a population of breeder reactors, and each individual breederreactor is an independent complex system within that population. TheRecord concept again reflects the knowledge of the system held by eitherpeople or computers. The breeder reactor may have its own Record ofitself stored in a computer that supervises its operations. The publicmedia and the Department of Energy will maintain other Records regardingthe plant. Any of these records may contain inaccuracies.

The Health State concept corresponds to a generic System State. As withHealth States, the System States often apply to specific parts of theplant. The Body Site concept corresponds to a generic PhysicalComponent, such as the plant (like the body), the core (like the heart),and pipes (like the throat, blood vessels, or urethra). Each SystemState will apply to some of these components. For instance, it may bereasonable to describe the integrity of any pipe by naming its SystemState from a group of Pipe leak states. Obviously, one pipe may beleaking or ruptured while another pipe is intact, exactly as we havefound with Health States occurring at Body Parts. A System State mightinclude an error in a supervising computer's code, leading the complexsystem to respond inappropriately in some situation. This roughlycorresponds to mental illness manifested by maladaptive behavior.

The Lead to relation again connects System States into parallelnetworks. Lead to relations again contain Modifiers which describeevents that make transitions between System States more or less rapid.For instance, an earthquake might cause a fatigued pipe to twist, leak,and finally rupture, just as a sports injury can cause an ankle tendonto stretch, tear, and finally rupture.

Findings again represent observable facts about the Complex System, suchas the temperature of a reactor's core, the water level of the core, orthe flow rate of water through a pipe. System States will be definedprimarily by the Specific Findings present. The exact Findings requiredwill be provided by a generation method, such as a Bayesian network thatreproduces experts logic about the clusters of Findings required toclassify a Physical Component of a Complex System as existing in aparticular System State. The simulation program asserts that the SystemState required for the simulation is present, then solves for allunknown nodes in this Bayesian network.

Courses of action again represent activities by humans, another externalsystem, or the system itself. Generally, these will be efforts torestore or maintain equilibrium of the system, or to intentionallyprepare the systems for a change of State. For instance, preparing thebreeder reactor for a scheduled shutdown and maintenance is a course ofaction similar to preparing a patient for surgery. Agents againrepresent inputs to the system that influence its Findings orprogression, such as cooling rods, water, fuel, or repairs to computercode.

Thus, we believe that the current invention has broad applicabilitybeyond the domain of medical simulations. It is especially likely to beuseful when the behavior of a system is so complex that an understandingof the system defies mathematical description. For instance, thisinvention is not well suited to simulating the flight of an airplane,which is fully described by physical laws. However, it might beexcellent for simulating maintenance of the airplane, which is likely toreflect obscure design decisions and even unknown, but empiricallyobserved, interactions between design decisions. Label in this Nuclearpower plant invention Generic concept example Population Complex systemNuclear power plant Record Record Press releases, DOE documentationHealth State System State Overheated core, Leaking pipe Body sitePhysical comp. Plant, Core, Pipe Lead To Lead to Intact pipe leads toLeaking pipe Modifier Modifier Bayesian network describing how age andEarthquake modify the pipe lead to Finding Finding Core temperature,Water level Gener. method Generation method Bayesian network describingan intact nuclear plant Course of Act. Course of Action Manual shutdown,automated shutdown Agent Agent Carbon rod, Water, Uranium, EarthquakeThe Empiric Simulation Program

The American Board of Family Practice (ABFP) is developing acomputer-based recertification process based on an Empiric SimulationProgram (ESP) that produces new cases from an editable knowledge base.This approach could yield practically endless numbers of cases at anaffordable cost per case, while maintaining a high level of security. Tomaintain an affordable cost per case, it is mandatory that the ESP notembed medical knowledge in code, that the ESP allows some stochasticvariation between cases, and that large chunks of the knowledge base arereusable. However, to identify acceptable performance, the system mustprovide means to compare a candidate's actions with those deemedappropriate by a relevant group of peers. This requires that theknowledge base store conditional logic.

Medical diagnosis and management is replete with conditional andprobabilistic logic that defies simple models or deterministicdescription. As an example, patents suffering from osteoarthritis (OA)may experience the disease starting early in life, following jointtrauma or other articular diseases, or late in life, especiallyfollowing years of excessive weight bearing. The rate of progression ofthe disease has not been described, but it can be slowed by weight loss.Knee joint destruction may occur in the lateral compartment, but morefrequently occurs in the medial compartment. Joint pain correlatespoorly with objective findings. Patients who seek care because of jointpain should usually be treated with acetaminophen first. Ifacetaminophen is inadequate, non-steroidal anti-inflammatory drugs canbe added or substituted. However, some of these drugs might acceleratejoint destruction. Furthermore, the entire class of drugs is morehazardous in the presence of gastric ulcers, renal disease,hypertension, bleeding disorders, and asthma. The gastric ulcer hazardcan be mitigated by either misoprostel or omeprazole, but probably notby H-2 blockers or sucralfate. Intra-articular steroid injections areuseful for acute exacerbations or if other therapies fail. Knee jointreplacement is an option, but replacement joints last about 10 years.Finally, the advent of new treatment options, such as COX-2 inhibitorsor injectable hyalin, could completely alter recommended care of OAalmost overnight.

Probabilistic and/or conditional logic thus pervades every aspect of OA,including incidence and prevalence, disease progression, physicalfindings, symptoms, recommenced management, and response to treatment.Nevertheless, the diagnosis and management of OA is relatively simplefrom a clinicians perspective. Many common problems are considerablymore complex. Consequently, we expect many medical problem domains tocontain at least as much conditional and probabilistic knowledge as theOA domain.

Early Scripting Efforts

The need for scripts was therefore recognized by Applicants at an earlystage of development. We had simultaneously begun to divide medicalknowledge into parallel networks comprises of health states (stages of awell described disease process) linked by “leads to” relations(describing the rate of progression from state to stage.) Health stateswere typically defined by asserting findings such as “normal height” or“normal hematocrit,” but the ESP would eventually have to provide anactual height and hematocrit. The range of normal values varies withage, sex and race. We also distinguished a concept of activities thatreveal data about a simulation (e.g. questions and lab tests), and aconcept of management criteria. Thus, we had five distinct concepts thatseemed to hold the vast majority of conditional logic: Health statedefinitions, Lead to descriptions, Specific Finding definitions,Revealing queries, and Plans for care. Scripts would be needed for eachof these.

Fortunately, we determined these scripts could be written to beindependent of any particular simulation, and therefore could bereusable. For instance, a health state definition could be completelyindependent of the process that lead to the health state. Whether OAdeveloped quickly because of overt trauma or slowly because of obesity,the same script would describe associated findings. Similarly, a scriptthat predicted the rate of progression could indicate that greaterweight produces faster progression, and that direct trauma produces veryfast progression.

Our first scripting approach was inspired by the scripting language ofThe Medical Record (TMR). TMR used scripts to inspect medical recordsand alert physicians to actions they might take. Those scripts typicallyconsisted of lines containing a conditional statement, an action to takeif the statement were true, and an action to take if the statement werefalse. The lines were executed in sequence, unless a GOTO statement sentthe program to a specified line. Logical loops, such as an instructionto vaccinate for tetanus every ten years, could be implemented usingGOTO statements. In our implementation, we developed an interpretedlanguage with a few standard queries to extract data from thesimulation, commands to write information from the knowledge base to thesimulation, and operators to manipulate information within the script.

This scripting language was rapidly implemented for the first prototypeof the simulator, demonstrated at the American Board of MedicalSpecialties meeting on computer-based testing in Chicago, Mar. 21-22,1996. It was easily extensible, but very difficult to write and almostimpossible to proofread or explain to physicians. In addition, theprocess of parsing and interpreting the script was an importantperformance bottleneck. In implementation, the scripts we created hadvery consistent logical flows related to their tasks. The scripts weremonotonously similar.

Sets of Conditions

The consistent scripting requirements allowed us to replace interpreteddata extracting queries and data writing commands with new classes ofobjects called Conditions. Condition subclasses currently includePerformance, describing an examinee's previous evaluations;FindingValue, for acquiring continuous values such as height orhematocrit; and Relational, for inspecting and establishingrelationships between patients and other entities. For instance, arelational condition may indicate that a patient Has a Health State, wasExposed to a disease causing Agent, or Exhibits a Specific Finding.

A Condition used as a query may return either a Boolean or continuousvalue. Relational and Performance Conditions typically yield Booleanvalues, in effect answering questions such as, “Has the patient had kneeOA for longer than 5 years?” FindingValue Conditions typically yieldcontinuous values, in effect answering questions such as, “What is thepatient's height now?” A Condition used as a command is a template forwriting new information to the simulation. These typically establishsome concept that persists until succeeded by another concept. Forinstance, a Condition would establish that a patient “Has glucoseintolerance starting now and lasting indefinitely.” This Condition wouldpersist until succeeded by the Condition that the patient “Has type IIdiabetes mellitus.”

We also designed a class called Sets to replace some of theprobabilistic information previously embedded in scripts. A Set containsseveral Conditions, and indicates how many ought to be present for theSet to be logically true. Sets support subset concepts such as exactly NConditions(N>O), between N and M Conditions(M>N≧0), and at least NConditions. Thus, Sets allow a succinct means of asking whether asimulated patient has any number of arthritic diseases, or askingwhether the patient has been prescribed exactly one non-steroidalanti-inflammatory drug, or stating that the patient must have at least 2and possibly four cartilaginous abnormalities.

Bayesian Networks

We implemented Conditions and Sets in a new model in 1996, but stillneeded a mechanism for organizing these concepts, and in particular forcreating dependencies. We could use Sets to define the state of a nodein a graph. We briefly experimented with using tree structures, in whicha tree node could have multiple states. Each state would be defined by aSet of Conditions, linked to other arbitrary information (such asmultimedia), assigned a probability, and point to another tree node. Touse a tree, the ESP would inspect the root node and determine whetherany of the states were already established or impossible in thesimulation. It would stochastically select one of the remaining possiblestates, then follow the corresponding branch of the tree, It wouldrepeat this process until it reached a terminal node. The terminal nodeand the path to it would provide the information the ESP needs to createa plausible patient, critique a physician's management strategy, orproduce a laboratory report conditioned on the nuances of a simulation.The ESP would perform these tasks in time proportional to the greatestdepth of the tree, or better.

Although the tree approach was technically feasible, many practicalproblems soon became evident. The first problem was the frequent need tonearly duplicate part of a branch with slight changes. For instance, theroot node in a tree that implements OA stages might inspect thesimulation for the current stage of OA, then produce findingsconditioned on that result. Each of its branches will describe jointspace narrowing in a probabilistic way, with some overlap. Mobility andpain nodes might depend on both the stage of OA and the joint spacenarrowing. Thus the tree has very redundant looking branches with onlyslight changes in probabilities. These trees are therefore hard toinspect.

Although Bayes nets are NP hard to solve precisely, they have severalwell known advantages. First, most of our node and state concepts couldbe reused immediately. Second, a Bayes net will almost never representthe same concept in two separate nodes; conditional probability tablesreplace the separate branches required in the tree structure. Third,other groups are actively developing analytic techniques that allow veryrapid approximation of Bayes net solutions. Fourth, companies such asNorsys Software Corp. (www.norsys.com) are developing affordablesoftware packages such a Neticarm™ and can provide a well documented,correctly functioning application programming interface to developers.Finally, Bayes nets can compactly represent very complicatedinformation, and allow knowledge editors, supervisors, and externalreviewers to interactively explore a model by setting the states ofnodes and inspecting updated probabilities.

Methods

The knowledge base was revised to accommodate Bayesian networks, forexample, represented as Neticarm™ files, node states defined with Setsof Conditions, and additional network and node details required by theESP. A production quality knowledge acquisition effort was initiated forOA and diabetes mellitus, and other diseases that predispose to orcomplicate these diseases. We began additional knowledge acquisitionefforts in otitis media, depression, hypothyroidism, abnormal Pap smearmanagement, and hypothyroidism.

The use of Bayesian networks as a scripting language has been partiallytested by implementation of an object oriented database which waspopulated with data about OA and obesity. Expressiveness was tested inthe other domains. We were actively programming an ESP that will relyalmost entirely on Bayes nets and Sets of Conditions to describeconditional and probabilistic information in family medicine, andpreparing to enter hypothyroidism data in an object oriented database.

Results

Current Structure

FIG. 19 illustrates classes in the relevant portion of the finalknowledge base structure. Lines indicate that one class is associatedwith another class. An asterisk, “*,” indicates a one to manyrelationship, while a number indicates an exact number of associations.For instance, a Health State is associated with 3 Bayes nets, whichprovide incidence, prevalence, and disease descriptions. Arrowsindicates ISA relationships. The lightly shaded classes require a meansof expressing probabilistic and conditional information, once providedby a scripting language.

The Bayes Net, Bayes Node, and State structures are replicated in partin the Netica™ file. This design requires that we maintain very strictname consistency between the knowledge base and Netica™ file. TheProperty may contain a time function (e.g., incidence or prevalence as afunction of age), a multimedia reference (e.g., a sound to play), simpletext, or a function of properties of other nodes. The seven mostimportant Relational Conditions are listed.

Expressive Verification

The knowledge acquisition effort has used these data structures torecord all of the conditional dependencies found to date. Spaceprohibits reproduction of Bayesian nets in this specification, but thefollowing fragments illustrate the use of Bayes nets and supportingstructures in place of scripts.

FIG. 20 illustrates a simplified OA generating Bayes net. The networkwas built as a roughly physiologic model of the development of OA,assuming that cartilaginous deformities and destruction cause jointspace narrowing, accompanied by sclerosis and subchondral cyst formation(not shown), and leading to gross deformities and loss of mobility. Painis a variable feature, but probably must be present in a test case(otherwise, how would the doctor's attention be drawn to the joint?).The mild narrowing state of the joint space node is defined by a Setcontaining one Condition, EXHIBITS the Specific Finding, mild jointspace narrowing, which is itself defined as a joint space of 4 to 6 mmfor the knee. The stage I state of the osteoarthritis stage node issimilarly defined by a Set containing one Condition, HAS mild OA. Theconditional probability tables for this node are arranged so thatcertain combinations of joint space narrowing and deformity define mildOA. Other combinations may define other stages, or be declaredimpossible (e.g. severe deformity without joint space narrowing might beimpossible in this context).

Two interesting benefits of this approach are first, that we can use asingle Bayes net to describe all five stages of OA, and second, throughthe logical magic of Bayes theorem, we can now invert a Bayes net builtfrom a perspective of classifying stages of OA, and use it to generatedescriptions of OA. We assert that the patient HAS any stage of OA,update probabilities throughout the network, and start stochasticallyassigning states to indeterminate nodes. With each assignment, we writenew information to the simulation, e.g., that the patient EXHIBITS mildjoint space narrowing. We can test the Bayes net by experimenting withit from both perspectives, e.g., beginning with an assumption ofcartilage damage to see what stage of OA results, or beginning with OAto see what other findings result.

Bayes nets supporting Leads to structures are conceptually very similarto Health State generating Bayes nets, with two important differences.First, the Conditions usually describe task factors for slower or morerapid progression, rather than features of a disease. For instance, anOA Lead To is likely to ask whether the patient HAS Obesity, or toassert that the patient is EXPOSED to some remedy.

Second, the goal of the Lead to structure is to produce a rate ofprogression, which is not specified anywhere else in the simulation. (Incontrast, the Health State generator has a goal of creating adescription consistent with an asserted disease).

FIG. 21 illustrates a Bayes net that could produce a report when anexaminee requests a Revealing x-ray test. The only simulation data usedin this report is the joint space, a value indirectly modified by theBayes net in FIG. 20. Now we have a continuous Bayes node acquiring itsvalue from a Condition that extracts the current joint space from thesimulation. Note that there are no requests to determine whether anySpecific Findings or Health States are present. Revealing queries shouldtherefore be reusable across simulations. Also, the accuracy of the testcan be built into the Bayes net representing that test. Another test,such as a magnetic resonance image, could have different size errors.

Subjective queries are much more complex, but still possible toconstruct using the same approach. The primary complication is thatsubjective responses are uniquely elaborate in temporal detail, yieldingstatements such as, “the pain has been coming going for weeks, but nowit is worse than ever.”

Management Plan critiques are similar to Reveals, except that most ofthe Conditions inspect prescriptions and queries made by the examinee,and the resulting report is a critique of a physicians' strategy. Ourexperience to date confirms the expectation that Bayes nets supportinferences about actions and plans.

Limitations

The current knowledge base design requires the additional validation ofsupporting a working simulator. We can not yet prove that the conceptspresented here will work in the simulator we are programming. Ourknowledge acquisition experience has uncovered one instance in which wethought that the data source for a continuous Bayes node (Hemoglobin AClevels) should be another Bayes network. This raised the specter ofsolving recursive (or accidentally cyclical) NP hard problems to producea simulation or answer a question. We expect that wary knowledge editingcan prevent such problems.

Bayesian networks, with appropriate supporting structures, are capableof representing important concepts in family medicine and seem likely toreplace other scripting options in a simulation program that we hopewill produce recertification tests in the future. We will soondemonstrate whether we are able to produce realistic simulations usingthis scripting language.

Computer-based testing holds promise as a technology that could addeducational content to the testing process while yielding different, andperhaps more important, information about examinees than paper-basedtests. Some computer-based tests use traditional multiple choice itemformats. Other tests simulate patient care experiences. Some elegantsimulation programs generate patient data from systems of equations, butmost outpatient medical problems still require empiric description. Someprograms embed the logic of the simulation in 2,3 code, although reuseand knowledge maintenance may be difficult.

The American Board of Family Practice (ABFP) is developing acomputer-based recertification process based on an editable knowledgebase. This empiric simulation project (ESP) could yield practicallyendless numbers of high quality cases at an affordable cost per case.Variability in case presentations should help the ABFP maintain a securetest. Conversely, modeling decisions which restrict the details of casehistories may reduce security.

The ESP development team designed an entity-relationship model ofmedical concepts and algorithms to create patient simulations from thedata model. These algorithms create patient histories and evolvepatients during simulated medical care. The central concept in thehistory generation algorithm is that patients with some health statesevolve to experience other health states.

The assumptions underlying early ESP algorithms and data models weresimilar to those of a Monte Carlo process. A simulated patient wouldhave partially completed a path through a Monte Carlo network. Aphysician's management decisions would influence the remainder of thepath. Nodes along this path represented the patient's overall healthduring a period of time, that is, all simultaneous medical problems arerepresented in a single Monte Carlo node. Arcs between nodes representthe patient's transitions between conglomerate health states. Othercommon decision modeling techniques, such as Markov processes anddecision trees, employ similar models of health states.

The Department of Family Medicine at Duke University and the affiliatedCabarrus Family Medicine Program conducted knowledge acquisitionexperiments for a variety of problems common in family practice. Theseincluded alcohol abuse, ankle sprains, diabetes mellitus, hypertension,osteoporosis, otitis media, peptic ulcer disease, pregnancy, reactiveairway disease, and smoking. These domains involve addictions andbehavioral problems, acute illness; acute illness superimposed onchronic predisposing illness; and non-systemic illnesses. The ESPdevelopment team advised the domain experts, and simultaneously modeledosteoarthritis of the knee and normal health.

These experiments demonstrated many serious difficulties with theconceptual model. First, to obtain variable histories required modelingmany nodes in a Monte Carlo simulation. In several domains a chronicprogressive systemic illness (e.g., osteoporosis) combined withrecurrent acute site-specific exacerbations or complications (e.g.,fractures of various bones). The original model implied the need for alarge number of conglomerate health states, for instance to definemultiple paths from “Normal health” to “Ex-smoker with severeosteoporosis and healed second left hip fracture.” The number ofconglomerate health states can expand quickly, and data required todefine these conglomerate health states (e.g., age specific incidence)is often speculative and redundant.

Second, identical information may be collected in several testingdomains. For instance, highly redundant obesity descriptions wouldappear in tests of osteoarthritis, diabetes, and hypertension.

Third, relations between health problems are unclear. Conglomeratehealth states do not compartmentalize disease processes, obscuringwhether domain experts consider hyperthyroidism or nicotine addiction asdirect precursors of osteoporosis, or risk factors, or distracters.

Fourth, modeling one therapeutic complication adds many nodes and arcs.Therapeutic complications are typically new illnesses superimposed onany of several antecedent conglomerate health states. For instance, apatient in any of the osteoporosis nodes might develop uterine cancerwhile taking unopposed estrogen. The number of nodes required in theMonte Carlo model may double, with an equal number of new arcs.Historical distracters, such as randomly appearing colds or a history ofappendicitis might require still more conglomerate states.

Finally, a computer-based test needs to specify the anatomy of disease,so that it can correctly present findings to the examinee. In somediseases the anatomy is erratic. A typical osteoarthritis patient willhave joints afflicted to different degrees.

Thus, Monte Carlo modeling techniques have an appealing ability togenerate multiple temporal sequences of events. However, the ABFP's needfor finer anatomic detail, reusable information, and manageableknowledge acquisition and maintenance required some revision of theMonte Carlo approach.

Methods

The ESP model was revised to define Parallel Networks of Health States,while discarding conglomerate health states. A Parallel Network includesa sequence of distinguishable, mutually exclusive Health States. Thesetypically reflect the medical literature's descriptions of stages ofprogression or severity of a disease. If the literature does not providea staging definition for a disease, Health States can usually be definedas absent, mild, moderate, and severe.

A parallel health state network connects these health states with “LeadsTo” objects, e.g., mild disease leads to moderate disease. A Leads Toobject associates specific collections of risk factors and treatmentswith a fuzzy rate of progression from the preceding to succeeding HealthStates. The risk factors may be Health States from other ParallelNetworks, activities (e.g., work, play, and habits), and family history.Treatments may be interventions prescribed by the examinee, or somesimulated previous provider.

Separate collections of Leads To objects manage history generation andevolution. In history generation, the ESP creates a life history andcontext for the examinee's encounter with the simulated patient. Theexaminer may want an unremarkable story compatible with many simulatedmedical problems, or a story that is virtually pathognomonic. Inevolution, an efficient test might routinely simulate rapid progressionof disease or complications of the examinee's treatments, regardless ofthe likelihood of these events in practice.

Each Parallel Network defined in a simulation imposes its Health Stateson one or more anatomic sites, which evolve simultaneously. Forinstance, a rheumatoid arthritis simulation could name a single ParallelNetwork and all of the joints affected. An osteoarthritis simulationmight use two copies of a knee osteoarthritis Parallel Network, applyingone to each knee. Different presenting Health States at each knee andindependent evolution of the knees would be typical of osteoarthritis.Systemic diseases involve the entire body of a simulated person.

Health States may recursively contain Parallel Networks representingmore acute exacerbations of the parent Health State. For instance,moderate osteoarthritis may include a Parallel Network describingtransitions between baseline and flare Health States. A simulatedpatient cycling between these Health States will display or recountepisodes of worsening arthritis symptoms.

The algorithms for history generation and evolution were adapted fromMonte Carlo techniques. A request for a simulation identifies thepresenting Health State in each Parallel Network. Using incidence andprevalence information, the age, sex and race of the simulated patientare selected. The time of the next (or, in history generation, thepreceding) event in each Parallel Network is predicted. In historygeneration, this may require assertions regarding the activities of thesimulated patient. The temporally closest event from all of the ParallelNetworks is instantiated. In history generation, the process ofpredicting the most recent preceding Health State change proceedsbackward through time until no further transitions are defined by theParallel Networks. In evolution, this process of predicting the nextevent continues until one of the events initiates another encounter withthe physician.

The revised ESP model was tested by additional knowledge acquisitionexperiments, implementation of a Poets object oriented database andsupporting algorithms, and generation of simulated osteoarthritis cases.The database was used to generate cases of osteoarthritis of the kneewith obesity as a risk factor, and gastric ulcers induced bynon-steroidal anti-inflammatory drugs prescribed without misoprostel.

Results

Knowledge Acquisition

Simple illustrations of their medical domains helped content expertsunderstand the scope of their knowledge acquisition tasks. Initiallyintricate domain models were decomposed into much less threateningParallel Networks. FIG. 22 illustrates common Parallel Networkstructures. The simplest network is a collection of one or more staticstates, typical of genetic (e.g., Downs syndrome) and some congenitalconditions (e.g., anencephaly). The progressive network is a series ofstates with no cycles, typical of degenerative illnesses such asosteoarthritis. The reversible network illustrates chronic butreversible conditions, such as essential hypertension and weightdisorders. In the injury network an acute insult evolves to eitherrecovery or a chronic condition with a later recovery. Injury networksdescribe many infectious diseases and trauma. The addiction networkillustrates that a person may abstain from, use, abuse, or becomeaddicted to a substance. In the scheme shown here, a previously addictedperson can only be addicted or recovering, but cannot return toabstinence, use or abuse. The surgical intervention overlay illustratesthat new states can be added to the above networks using irreversibletherapies such as radiation or surgery. Domain experts adapted thesenetworks to their needs by eliminating unwanted nodes and arcs, orreplacing nodes with another network.

Domain experts began with a primary Parallel Network to sketch thediseases defining their domain, such as stages of diabetes mellitus.Parallel Networks of comorbid conditions were identified in mostdomains, typically including risk factors for progression through theprimary network, such as obesity. Most domains included one recursivelayer of Parallel Networks representing exacerbations of Health Statesin the primary Parallel Network. Most domains also identified one ormore Parallel Network representing complications of Health States in theprimary network, such as retinopathy, or of treatment of primary HealthStates, such as gastric ulcers.

Experts were asked to estimate 1) how long a risk factor should existbefore it could influence a transition between states in a primarynetwork, 2) the time required for transitions in the primary network,given different combinations of risk factors, and 3) the number ofpasses an individual patient should be allowed to make through a cycle(e.g. from acute injury to recovery and back). Although these data wereoften non-existent in the literature, domain experts could comfortablyestimate a range of values from clinical experience. Although the datato gather remained imposing in volume and dauntingly quantitative,Parallel Networks in the revised ESP model appeared to successfullyguide segmentation of data into intellectually plausible sets.

Data Model and Algorithm Implementation

The osteoarthritis experiment continued with development of an objectoriented database structured after the ESP model. The database waspopulated with information about four stages of osteoarthritis, threeweight conditions, and 2 ulcer states.

The algorithms mentioned above were implemented, but without support foracute exacerbations or multiple Parallel Network copies afflictingdifferent anatomic sites. Conditional probabilities were managed with asimple scripting language. The scripting language has since beenreplaced by Bayesian networks.

Instantiation of the model confirmed the expected difficulty inauthoring a family of cases with the same underlying disease process,but different details in presentation. In particular, giving attentionto conditional probabilities slows knowledge acquisition considerably.Memories of individual clinical cases were helpful in authoring anarrowly defined simulation, but much more attention was required toproduce Health States generation methods and Leads To objects that wererobust to changing assumptions about sex, race, and obesity. In spite ofthese difficulties, data entry in a data base founded on ParallelNetworks was accomplished.

Experimental Verification

The prototype ESP simulator generated a series of patients fordemonstration at the American Board of Medical Specialties meeting oncomputer-based testing in Chicago, Mar. 21-22, 1996. Approximately 30patients were generated and stored over a four day period, includingseveral during the meeting. Each patient generation required about 20minutes. After generating a variety of male and female patients, data inthe knowledge base were skewed to generate middle aged overweight whitefemales. These patients were typically 55 to 65 years old and complainedof recently worsening pain in one or both knees. Patients had beenmorbidly obese for 1 to 3 years prior to presentation, and had at leasta 5 year history of mild arthritis in the affected knees.

Their health problems began with either obesity or mild osteoarthritis10 to 30 years prior to presentation.

During the demonstration, most history and laboratory requests returnedgraphs of values over the simulated patient's lifetime, enabling viewersto see how variables such as weight, uric acid, or osteophyte numbershad changed since birth. These graphs demonstrated concurrent historiesof worsening osteoarthritis and obesity.

Demonstration patients were managed interactively. Patients managed withhigh doses of nonsteroidal anti-inflammatory drugs without misoprostelwould develop ulcers sometime during a 2 year follow up period. Weightloss was also possible. Optimal management of weight and prescription ofstrengthening exercises would slow the inexorable progression of kneeosteoarthritis, but progression from moderate to severe knee arthritiswould inevitably occur within 10 years.

Discussion

The simulations demonstrated that the prototype system could generatepatients with plausible medical histories; appropriate symptoms, signs,and laboratory values; and could evolve patients over time. Theseparation of data controlling osteoarthritis, obesity, and ulcerhistories and presentations suggests that these components would bereusable with modest modification, if any, in new disease domains.Substantially different osteoarthritis simulations could be produced byreplacing a few history controlling Lead To objects.

Limitations

We are currently developing a simulator with acute exacerbations, pastmedical interventions, and use of multiple copies of one ParallelNetwork's data. The new model and algorithms replace simple scripts withBayesian networks. Although the next generation simulator is not yetfunctional, no fatal conceptual difficulty is evident.

The knowledge acquisition problem for the ESP model remains daunting.One vexing problem is that the history generation algorithms requiresolutions to multiple temporal constraints. These constraints may notalways have a solution, and it is not yet clear how to react if ahistory generating step fails, or how to guarantee temporal solutionswhile reusing data.

The Cartesian product of N parallel networks creates an N dimensionalgrid whose nodes represent conglomerate health states. This grid is acomplex Monte Carlo model with many low probability paths that wouldnever have been considered in an explicit Monte Carlo model. Conditionalprobabilities within Parallel Network's Leads To objects could provide ameans of pruning the N-dimensional space. This mechanism may not work,as it places further burdens on knowledge acquisition and reusableobject design.

These limitations must be considered in context. In the absence ofmathematical models of the diseases of interest, the ABFP requirementsfor secure tests, realistic temporal and clinical features, anddefensible credentialing decisions, complex data is an inevitablefeature of a computerized problem generation process.

Parallel Networks facilitate some aspects of knowledge acquisition for apatient simulation knowledge base, and appropriate algorithms supportgeneration of patients. The data required are relatively reusable, incontrast to data explicitly describing global health. Furtherexperimentation is required to demonstrate that this approach remainstractable with more complex scenarios. Parallel Networks may haveapplication in other endeavors that traditionally describe globalhealth, such as decision analysis.

Background Information for Knowledge Development

Medical certifying organizations have traditionally relied upon paperand pencil cognitive examinations to measure certification candidates'suitability for board certification. Traditional formats such asmultiple choice questions have well-defined operating characteristicsand reliability for examining cognitive knowledge capabilities. Theyprovide, however, only primitive ability to assess a candidate'sproblem-solving capabilities. Additionally, traditional testingstrategies rely upon a continuous process of item development; onceused, the items in an examination must be replaced with new questions inorder to preserve security of the certification process. Eachexamination represents a product that the certifying organization canuse only once. The presently used medical certification process thussuffers from two weaknesses: 1) test development requires re-generatingan examination with new material on a recurring (usually annual) basis;2) although multiple choice questions demonstrate reliable performancein measuring cognitive knowledge, this format doesn't measure adequatelyproblem-solving capabilities.

Several organizations have experimented with computer-delivery ofclinical content and evaluation. In the late '60's and 70's, the OhioState University developed a self-directed Independent Study Programthat utilized a “Tutorial Evaluation System” or TES for conveyingcurriculum content. About the same time, Dr. Octo Bamett's laboratory atthe Massachusetts General Hospital began development of clinicalsimulations using the MUMPS language. Investigators at the University ofIllinois developed a simulation model known as Computer AssociatedSimulation of the Clinical Encounter, or “CASE’). Research supported bythe American Board of Internal Medicine demonstrated that a computerizedexamination system appeared feasible in professionalevaluation/certification settings. Stevens and colleagues alsodemonstrated the feasibility of using computer-based systems for testingproblem-solving ability in undergraduate medical school curriculumapplications. Additionally, Sittig and colleagues examined the utilityof computer-based instruction in teaching native users basic computertechniques such as “drag and drop” and other computer operations. Theseefforts suggest that computer-based testing techniques will similarlytransport to the computer-native medical certification candidate.

Another system with special relevance to ABFP's efforts was developed atthe University of Wisconsin. This project served as the nidus for theComputer-Based Examination (CBX) developed by the National Board ofMedical Examiners (NBME). NBME's CBX development project has been inevolution for over a decade, and has demonstrated validity in examiningprofessional degree candidates. The CBX development experience suggeststhat clinical computer simulations with automated scoring algorithms canproduce professional certification examinations at reduced cost comparedto traditional methods. However, the CBX model suffers from one majordrawback: the clinical simulations are “hard-wired” in computer sourcecode which must be re-coded for each new examination. Once thesimulation has been used widely, the examination contents are no longersecure, necessitating continuous cycles of new simulation development.

To circumvent these weaknesses, ABFP embarked upon a computer-basedtesting project which will 1) generate new patient cases for eachcandidate examined, and 2) test a candidate's problem-solving ability.The system relies upon a knowledge base of family practice thatrepresents in probabilistic terms disease/condition incidence,prevalence, evolution over time, and response to interventions.

Discussions with other certification organizations (other specialtyboards, professional organizations) have emphasized the potential needand market for knowledge-based systems in training and evaluationcontexts. The expert system literature affirms this evolution inevaluation and training systems. Early artificial intelligence/expertsystem work concentrated on “rules of thumb” or heuristics to representproblem-solving strategies identified by domain experts. Instances ofthese rule-based systems demonstrated that they were necessarilyconstrained to narrow domains, and that the knowledge contained in therules was difficult to validate. Research has also indicated thatexperts relate only one dimension of knowledge when defining a rule, butalso rely upon expansive knowledge of how systems work (i.e., physiologyand pathophysiology in the medical domain) in performing real-worldproblem-solving. This realization has led to re-thinking regardingstructure of knowledge-based systems to reflect the tasks such a systemshould accomplish, the methods the system should use to accomplish thetasks, and the knowledge required to support these methods.

Knowledge-acquisition for such systems entails development of a modelfor the domain and instantiation (ie, encoding and enter neededinformation into the system's data structure) of the model withinformation acquired from knowledge “donors”. The model structurenecessarily drives the knowledge acquisition effort. ABFP's computerbased testing system under development at ATL while not an expert systemper se, represents knowledge at multiple levels of complexity. Forexample, reactive airways disease is represented as a series of healthstates: Normal (Non-reactive) Airways, Reactive Airways-Mild, ReactiveAirways-Moderate, and Reactive Airways-Severe. Each health statecontains identifiers which relate the particular health state toprecedents and antecedents (eg, Normal Airways serves as the precursorhealth state for Reactive Airways-Mild which precedes ReactiveAirways-Moderate, which in turn leads to Reactive Airways-Severe.) Eachhealth state in turn is associated with specific versions of universallyobservable findings. For example, a Finding called “Asthma AttackFrequency” is universally observable, although most people enjoy aNormal Airways health state and its associated frequency of asthmaattacks of No Attacks (e.g. a Specific Finding indicating 0attacks/month, indefinitely). Similarly, the Finding “Shortness ofBreath” is instantiated with the Specific Finding “No shortness ofbreath” in the Normal Airways state. Likewise, other Findings such asRespiratory Function and Severe Asthma Attack Frequency are instantiatedwith corresponding normal Specific Findings (Normal RespiratoryFunctions, and No Severe Attacks.) This representation of Findings withHealth State-specific instances of Specific Findings provides re-usablestructure which transports to each new health state. Such reusabilityhas been identified as a characteristic which contributes to therobustness of a knowledge-based system.

Another example will illustrate further the relationship betweenFindings and Specific Findings. Consider the growth curve charts we usein assessing child development. A Finding associated with growth isHeight, a universal property of individuals. Normal Height wouldrepresent a Specific Finding which describes the range of heightsassociated with normal growth. Growth charts for Boys and Girls wouldthen be described as Patterns which define the normal probabilitydistributions for growth in boys and girls. At the start of asimulation, a percentile (e.g., 29th percentile) would be selected forthe patient's growth characteristics. Then a Pattern for the particularpatient is instantiated using the 29th percentile curve from theappropriate gender growth chart. How does the examinee learn about thepatient's height? A Reveal Course of Action (COA) is initiated (themechanics of this aren't important to understand at this point) toobtain the patient's current value for Height Finding.

Simmons and Davis have identified the importance of the distinctionbetween actual knowledge and representation of knowledge. Knowledgedescribes the attributes of a health state; representation consists ofthe symbols and language used to encode the knowledge in the testing orexpert system. Sinunons and Davis have additionally identified thatknowledge of multiple types is needed for robust performance. Theseauthors have partitioned knowledge into three fundamental types:knowledge about tasks, knowledge about methods, and knowledge aboutmodels of system behavior. These types correspond to those included inATI's Computer based Testing system. Findings, Specific Findings,Patterns and Sub-pattens describe system behaviors and characteristics.Courses-of Action describe tasks and methods used to apply, modify, andevaluate the health state information and characteristics described inthe model. As also indicated by Simmons and Davis, subdivision ofknowledge types in this manner facilitates the knowledge acquisitionprocess. This subdivision also promotes multiple levels of knowledgeabstraction, which enhances the system's ability to represent varyinglevels of complexity. For example, in the Computer-based Testing system,a Pattern such as incidence is further subdivided into sub-pattens suchas incidence in females versus males, and incidence in variousracial/ethnic groups.

To facilitate development of such a system, the developers divided thesystem development task into three components: the knowledge base, thepatient simulation generator, and the presentation system.

The knowledge base has been designed and represented as a series ofentity-relationships. The model has several fundamental entities:Patient, Health States, Findings, Courses of Action (COA), and Agents.These entities have relationships of INTERACTS-WITH, CONTACTS,IS-RELATED, EXHIBITS, HAS, EXPOSED-TO, LEADS-TO, ASSOC-WITH, LINKS_TO,USES, IDENTIFY, MANAGE, ALTER, REVEAL, and EVALUATE.

Referring to FIG. 1, which describes the entities and relationshipsincluded in the model, rectangles indicate relationships betweenentities in the model. Hexagons indicate entities. Solid lines indicateMedical Knowledge Relationships (e.g., a course of action such astreatment with non-steroidal anti-inflammatory agents can modifySpecific Findings such as pain in the patient with osteoarthritis).Dotted lines indicate Simulation/Evolution relationships which definehow a particular domain simulation can proceed.

The model depicted in the figure has been published in the Journal ofthe American Board of Family Practice, and presented to nationalaudiences.

The patient simulation generator relies upon a series of generationmethods to create patients for presentation to thecertification/recertification candidate. These algorithms function toevolve the patient forward (to reflect progression of the diseaseprocess and response to interventions) and backward in time (to create apast history for the patient.)

The patterns which describe patient progress and characteristics aredefined as probability distributions (discrete and continuous asappropriate for particular finding) during the knowledge acquisitionprocess. At the beginning of a simulation, a random number generatorselects a master percentile” (MP) which then serves as the reference forselecting particular patterns from the appropriate specifieddistributions. Properties of these patterns are then presented to thecandidate as findings for a particular health state (e.g., the currentglucose level as a manifestation of diabetes.) Once presented with thepatient description (age, race, gender, clinical findings), thecandidate then selects appropriate COA's for further evaluation and/ormanagement of the patient's health state.

Selection of an interventional COA disturbs the simulation in one orboth of two ways. First, it may cause the simulated patient to changehealth states, e.g., by removing a appendix, the patient may proceedfrom a health state called “appendicitis” to one called “postappendectomy.” Second, the intervention may initiate a patternrepresenting a temporary perturbation of some finding. For instance,administering acetaminophen to a febrile otitis media patient may notcause a change in health state—an infection may still exist, and thefever it induces will return in a few hours. Nevertheless, acetaminophenadministration will reduce the fever for a short time. This perturbationmight be represented in the knowledge base in the Temperature Finding,Fever Specific Finding, Antipyretic perturbation, with a four hourduration temperature change initiated by any antipyretic. When theexaminee requests a temperature, the COA that reveals the currenttemperature must combine information about the underlying temperatureand all antipyretic drugs administered in the last four hours.

The distinction between changing health states and perturbing findingsis necessarily artificial (health states are just collections offindings), and the decision to model a particular process one way oranother may often depend on testing goals, and subsequent decisionsabout how finely to model health states. In general, very finedistinctions between health states should result in more interventionsthat change health states, while coarsely defined health states mayrequire more perturbations in Findings.

A COA can modify the health state in which a patient exists at one pointin time. When the candidate selects such a COA, the simulated patientmay evolve to a new health state on the basis of patterns specified forhealth state evolution in the knowledge base. The knowledge for aparticular health domain is stored as a parallel health state network.For example, the initially generated patient for a case ofosteoarthritis will demonstrate some stage of osteoarthritis. However,other health states such as obesity might influence the progress of thepatient's arthritis from mild to moderate and moderate to severedisease. In the parallel networks of health states representation, anewly-generated patient will display findings consistent with a healthstate in the primary domain (for example, osteoarthritis) and in theparallel health states (e.g., obesity) which influence the primaryhealth state's progress. As shown in the following figure,osteoarthritis can progress over time from the normal state to mild,moderate or severe osteoarthritis.

For this particular illness, progress occurs in one direction only;osteoarthritis doesn't regress once developed, but can stabilize at aparticular degree of severity. Obesity represents a parallel healthstate which can influence the progression of osteoarthritis. Mild,moderate, and severe obesity can influence this progress at differentrates: the model permits representation of greater impact for moresevere obesity states. Notice also that obesity can regress (severeobesity can revert to moderate obesity, etc.) Similarly, other parallelhealth states might exist which could modify progression ofosteoarthritis. For example, the patient who has osteoarthritis willfrequently utilize nonsteroidal anti-inflammatory drugs (NSAID's) fortreatment. These agents can improve the symptoms of osteoarthritis, butalso impact on the parallel state of peptic ulcer disease, ie treatmentwith NSAID's can induce an ulcer, which will then evolve in parallelwith the course and treatment of osteoarthritis. Initial experience withthis representation indicates that these modifier-relationships are notwell-defined in the medical literature and constitute a research areafor further development.

The simulation system's fidelity depends upon access to a richrepresentation of health state-specific knowledge. This knowledgeconsists of Findings obtained from physician “knowledge-donors” workingfrom templates provided by the Assessment Technologies, Inc. developmentteam. The template includes a NAME for the health state and anassociated SNOMED code. The template also includes specific descriptionsof the Findings, and Patterns for these Findings. The patterns arestored as distributions; these distributions are obtained from themedical literature where available, and from physician expert opinionwhere such published data don't exist.

The development team doesn't expect the knowledge groups to providethese distributions but rather to indicate the relationships betweenhealth states, how parallel states influence each other qualitatively(e.g., increase, decrease, or stay the same), and possible sources ofinformation about the relevant probabilities.

The knowledge model has evolved over the past six months to includeextensive use of belief networks (also called Bayesian networks). Beliefnetworks provide a graphical process for describing the relationshipsbetween entities in a health state. For example, some set ofcharacteristics (family history, age, gender, racial origin/ethnicity,body weight) influence the development of impaired glucose tolerance.

FIG. 24 illustrates these relationships: Family History, Gender, Race,Weight, and Age, all of which influence the development of impairedglucose tolerance. The raw pictorial doesn't say how they influence IGT,but rather that they “influence” the development of IGT. In thebackground, we incorporate probabilistic information which describesthese relationships quantitatively, but would expect the knowledgedevelopment group to provide only semiquantitative guidance (e.g., aperson whose mother has diabetes has twice the likelihood of developingIGT compared to an individual who has no such family history.) We intendto fill in the more specific quantitative probabilities on the basis ofdata in the literature where available; if such information does notexist, we will have to rely on expert opinion.

How the Knowledge Development Process will Work

What will we need from the knowledge team in order to generate theinformation required in our system? The team should proceed in astep-wise fashion to address the following issues:

-   -   How is the health state defined? (e.g., What criteria do we use        to define the presence of impaired glucose tolerance or diabetes        mellitus?)    -   What population/s do/es the condition affect (should the system        emphasize a particular population group?)    -   What are the commonly accepted stages of the disease process?        What demographic/patient characteristics, risk factors, and        behaviors influence a patient's movement from one stage to        another? (e.g., obesity's influence on hypertension and        diabetes.)    -   How do particular characteristics vary within a given stage of        illness (e.g., what blood pressure ranges would we expect in        Stage 1, 11, etc., for hypertension) How should these        relationships appear in the Bayesian network format? What        therapeutic modalities (pharmacologic, nonpharmacologic) exist        to modify the progression and/or severity of the disease        process? (e.g., magnitude of effect of weight loss on blood        pressure in Stage I, Stage 11, etc; how much will weight loss        lower blood pressure? How much will weight loss decrease the        likelihood of progressing from Stage I to Stage 11        hypertension?)    -   What guidelines exist to describe optimal management plans for        the disease process (e.g., JNC VI for hypertension)    -   For a given health state, what management and diagnostic        concepts should we emphasize in creating the knowledge base (we        cannot reproduce Harrison's in the knowledge base, nor should        we—the system has to be good enough, not necessarily exhaustive)    -   What parallel health states should the model reflect for the        primary disorder in questions (e.g., for osteoarthritis, obesity        and peptic ulcer disease might affect disease progress and        treatment, respectively)    -   What multimedia resources will we need to represent adequately        the clinical findings associated with the health states?        How is the health state defined?

The group should identify the criteria (physiologic, clinical,demographic, etc) which define the disease process, and whichdistinguish the various stages of the disorder. To whatever extentpossible, we should rely on nationally accepted criteria as published inthe peer-review literature and highly regarded textbooks.

What populations does the condition affect (should the system emphasizea Particular population group?)

We might encounter health states or diseases for which the ABFP wants toemphasize how the disorder affects certain groups. We will attempt tohave one family physician ABFP director on each of the knowledge teamsto provide the Board's perspective in this regard.

What are the commonly accepted states of the disease process?

The ABFP has used for many years the Disease Staging system produced bySystemetrics (originally developed at Jefferson Medical College). Fordisease processes which don't have commonly accepted staging criteria,we should use the Systemetrics system. However, for those disorderswhich have nationally accepted staging criteria, we should use thoseinstead. For example, the INC VI has described the following stages forhypertension: Optimal, Normal, High-normal, Stage 1, Stage 2, and Stage3.

What demographic/patient characteristics, risk factors, and behaviorsinfluence a patient's movement from one stage to another? (e.g.,obesity's influence on hypertension and diabetes)?

Issues such as age, gender, family history, body habitus, behaviors,occupational exposures, etc, affect the likelihood that an individual'sdisease will progress (or regress) from one stage to another. We willneed information regarding 1) what are the important risk factors, 2)what is the magnitude of these factors' impact on the disorder'sprogress, and 3) what is the approximate time frame for these changes?

How do particular characteristics vary within a given stage of illness(e.g., what blood pressure ranges would we expect in Stage I, II, etc.,for hypertension)?

This relates to the stage descriptions alluded to above; however,individuals within a given disease stage will also exhibit somevariability, ie patients within Stage I of hypertension will demonstratea frequency distribution of systolic and diastolic blood pressureswithin the stage definition. These values might define normaldistributions, uniform distributions, or some totally skewed dispersion.The group might not know the exact shape of these curves, but, to theextent possible, should indicate qualitatively what generalconfiguration we should anticipate. Staff at A.T.I. will generate thesedistributions from literature sources.

How should these relationships appear in the Bayesian network format?

As noted earlier, the model uses Bayesian networks extensively to depictrelationships, effects of therapy, progression of disease, choice oftherapy, calculation of drug doses, and results of diagnostic testing.As the group identifies health states, the members should enumerate thedemographic characteristics which influence state transitions,attributes which influence selection of therapy, etc. Each diseasedomain will include literally dozens of these structures. Although theteam will not be expected to construct and develop fully each of thesenetworks, the knowledge engineer will need guidance in what networks todevelop. Again, we do not expect the team members to become facile inthe creation of these networks; however, once developed, the team willhave the opportunity to observe the behavior of these structures toconfirm that they behave as intended.

What therapeutic modalities (pharmacologic, nonpharmacologic) exist tomodify the progression and/or severity of the disease process? (e.g.,magnitude of effect of weight loss on blood pressure in Stage I, StageII. etc, how much will weight loss lower blood pressure? How much willweight loss decrease the likelihood of progressing from Stage I to StageII hypertension?)

We will need information regarding optimal recommended therapies,pharmacologic and nonpharmacologic, which we would expect familyphysicians to employ in managing the particular health state.Additionally, we need some indication about how the therapy affects thedisease process. For example, does weight loss decrease blood pressure,and by how much (large, moderate, small amount)?

What Guidelines exist to describe optimal management plans for thedisease process (e.g., JNC VI for hypertensions?

To whatever extent possible, we want the system to reflect well-done andbroadly-accepted clinical guidelines. For some of the domains, no suchdocuments exist and we will have to create our own “guideline” as wedevelop the health state. For others, such as hypertension, fairlyextensive and accepted guidelines exist (e.g., JNC VI), and the systemshould reflect these guidelines as closely as possible. Also, we shouldattempt to utilize ABFP reference guides for those domains for which theBoard has produced these documents.

For a given health state, what management and diagnostic concepts shouldwe emphasize in creating the knowledge base (we can't reproduceHarrison's in the knowledge base, nor should we—the system has to begood enough, not necessarily exhaustive)?

We will attempt to include a family physician ABFP Board of Directorsmember on each team to provide Board input into health state emphases.In developing diabetes mellitus and osteoarthritis, we frequently findourselves saying, “That's an issue for ABFP to decide.” For example, inassessing a candidate's ability to manage osteoarthritis of the hand, dowe want to investigate the candidate's ability to interpret a handradiograph, or rather do we want to know how the candidate uses theinformation that a patient's x-ray demonstrates stigmata ofosteoarthritis? The first question deals with a psychomotor skill(radiograph interpretation), while the second question assesses thecandidate's cognitive knowledge regarding therapy for osteoarthritis.Having an ABFP Board member on each committee should help provide ABFPinput into such decisions. Nevertheless, committee member input may behighly valuable to the Board, and we encourage members to contemplatethese issues: What are the critical commissions and omissions in careplans for these patients? What are the simplest approaches to improvinglength and quality of life? What are the common mistakes in clinicalcare? What are the new insights into appropriate clinical care? What arelikely to be the testable concepts related to this health state domain?

What parallel health states should the model reflect for the primarydisorder in question?

For osteoarthritis, what other health conditions might influence theprogress and/or management of the arthritis? For example, obesitycertainly has an impact on the progress of osteoarthritis. Additionally,the presence of peptic ulcer disease will have a substantial impact ontherapeutic options. Extended use of NSAID's could influence renalfunction. In this context, obesity, peptic ulcer disease, and renalfunction represent parallel health states: conditions which coexist andinteract with osteoarthritis.

What multimedia resources will we need to represent adequately theclinical findings associated with the health staters?

One of the advantages of computer-based testing is the ability topresent a variety of media to the candidate: sound, video,still-photographs and graphics can all enhance the system's appearanceand provide the ability to assess psychomotor skills in real time. Foreach of the health states, what media should we acquire and how shouldwe use these resources? For example, as we develop a module on heartfailure, do we need third heart sounds and chest x-rays? Media representa typical cost-effectiveness question in assessment: these resourcescost substantial amounts. Does the information gained from presentingthe media justify the acquisition cost? There's no easy answer to this,but we need to keep in mind that, for the most part, we will have topurchase a lot of this material. Obviously, we can use photographscurrently in an item bank. Optionally, video and sounds come fromexternal sources. Are there testable concepts that require a physicalmodel to test in sufficient detail (performing a sigmoidoscopicexamination, suturing)?}

Further, as indicated herein, the present invention may be appliedacross a broad range of programming languages that utilize similarconcepts as described herein. The present invention may also be used ina distributed environment/architecture, optionally using thin clienttechnology.

FIG. 21 is an illustration of the architecture of the combined internet,POTS, and ADSL architecture for use in the present invention inaccordance with another embodiment. In FIG. 21, to preserve POTS and toprevent a fault in the ADSL equipment 254, 256 from compromising analogvoice traffic 226, 296 the voice part of the spectrum (the lowest 4 kHz)is optionally separated from the rest by a passive filter, called POTSsplitter 258, 260. The rest of the available bandwidth—from about 10 kHzto 1 MHz—carries data at rates up to 6 bits per second for every hertzof bandwidth from data equipment 262, 264, 294. The ADSL equipment 256then has access to a number of destinations including significantly theInternet 268, and other destinations 270, 272.

To exploit the higher frequencies, ADSL makes use of advanced modulationtechniques, of which the best known is the discrete multitone (DMT)technology. As its name implies, ADSL transmits data asymmetrically—atdifferent rates upstream toward the central office 252 and downstreamtoward the subscriber 250.

Cable television providers are providing analogous Internet service toPC users over their TV cable systems by means of special cable modems.Such modems are capable of transmitting up to 30 Mb/s over hybridfiber/coas systems, which use fiber to bring signals to a neighborhoodand coax to distribute it to individual subscribers.

Cable modems come in many forms. Most create a downstream data streamabove 50 MHz (and most likely 550 MHz) and carve an upstream channel outof the 5-50-MHz band, which is currently unused. Using 64-statequadrature amplitude modulation (64 QAM), a downstream channel canrealistically transmit about 30 Mb/s (the oft-quoted lower speed of 10Mb/s refers to PC rates associated with Ethernet connections). Upstreamrates differ considerably from vendor to vendor, but good hybridfiber/coax systems can deliver upstream speeds of a few megabits persecond. Thus, like ADSL, cable modems transmit much more informationdownstream than upstream.

The internet architecture 220 and ADSL architecture 254, 256 may also becombined with, for example, user networds 222, 224, and 228. Asillustrated in this embodiment, users may access or use or participatein the administration, management computer assisted program in computer240 via various different access methods. In this embodiment, thevarious databases 230, 232, 234, 236 and/or 238 are accessible viaaccess to and/or by computer system 240, and or via internet/local areanetwork 220. These databases may optionally include objective criteriafor evaluating the corporate governance characteristics for ranking thecorporation.

For example, environmental data is generally publicly available whichindicates a corporation's compliance history, outstanding violations orpotential violations, and the like. Similarly, standard legal and/orregulatory and/or administrative and/or business databases may beconsulted to obtain additional information on corporate governancetechniques, potential for government intervention, shareholderparticipation and/or customer loyalty. All this data may then becollected and analyzed to determine the overall attributes of thecorporate, shareholder, government, and customer agents, for input intothe simulation. Alternatively, the individual data may be used and inputinto the simulation, and the simulation may digest or process the dataindividually or collectively as part of the simulation.

In accordance with this embodiment, workstation 240 optionally includesmodules 242, 246, 248, and 250 for individually handling theoperations/simulation of the different agents. Alternatively, one moduleor a different number of modules may be used for processing the agentrelationships, processes, and or interactions.

Alternatively, users may access or use or participate in the simulationprogram for decision making, indexing, ranking, and the like, viavarious different access methods as well. The above embodiments are onlyto be construed as examples of the various different types of computersystems that may be utilized in connection with the computer assistedand/or implemented process for decision making, indexing, ranking, withrespect to corporate governance.

Of course, another result of the simulation is identifying companies forinvestment purposes, and actually investing in these companies. Further,the actual investments may be done manually and/or electronically, andoptionally over the internet.

According to another embodiment, the instant invention includes a methodfor evaluating or educating a user, such as a physician, for example, asshown in FIG. 25. In Step S100, parallel health state networks, forexample, describing disease evolution in various medical/domains orhealth states are generated by a computer or a user. In Step S110, aknowledge base is scripted by a user or a computer using belief networksand/or causal probabilistic networks, such as Bayesian networks, andbased, at least in part, on the generated parallel health statenetworks. In step S120, the computer instantiates a model or virtualpatient, at least in part, from the scripted knowledge base and displaysto the user one or more non-normal health states of the model or virtualpatient. In step S130, the user inputs one or more courses of action toaddress the one or more non-normal health states. Alternatively, theuser inputs a query to the computer for a specific medical finding, forthe patient such as would be obtained by running a medical test orexamination on the patient. In such an event, the computer provides tothe user the requested medical finding and returns to step S130. In stepS140, the computer evolves the model or virtual patient based, at leastin part, on the generated parallel health state networks and the userinput. In step S150, the computer determines whether the user hascompleted treatment of te patient. if not, method flow returns to stepS130. Otherwise, in step S160, the user's inputted courses of action andqueries are evaluated relative to accepted norms of medical practice bythe computer.

By way of example, FIG. 26 shows additional or alternative steps forevaluating a user. In step S200, decision variables, which can becontrolled, such as courses of action, and utility variables, which areto be optimized, such as health states of medical domain in parallelnetworks are generated by a user or a computer. In step S210, a decisionnetwork based on the belief networks or causal probability networks, thedecision variables, and the utility variables, is generated to determinean optimum treatment for the instantiated model or virtual patient. Instep S220, the computer compares the generated optimum treatment withthe courses of action and queries inputted by the user. The method flowthen proceeds to step 150 as discussed above.

In another embodiment, the instant invention includes an expert system.By way of example, the expert system is a stand-alone unit.Alternatively, as shown by way of example in FIG. 21, the expert systemis communicatable with a user via a computer network. The computernetwork includes, for example, POTS for a dial-up expert system and/orthe Internet or WAN for a Web-accessible expert system.

In one embodiment of the expert system, the instant invention includes aprogram having instructions for executing the expert system. By way ofexample, in Instruction S300, the computer receives patient data for anactual by user input. In Instruction S310, the computer instantiates avirtual patient having characteristics consistent with the receivedpatient data and based, at least in part, on one or more belief networksand/or causal probabilistic networks describing disease or health stateevolution. In Instruction S320, the computer generates a query to theuser for a specific medical finding concerning the actual patient, or acourse of action based, at least in part, on the instantiated virtualpatient and the one or more belief networks or causal probabilisticnetworks. In Instruction S330, the computer receives the specificmedical finding from the user responsive to the generated query. InInstruction S340, the computer evolves the instantiated virtual patientin accordance with the above-mentioned belief networks and/or causalnetworks and the received specific medical finding and/or the generatedcourse of action. In Instruction S350, the computer determines whetherthe user has dispensed complete treatment of the actual patient based,at least in part, on the generated courses of action, for example, for agiven medical visit or encounter. If not, method flow is returned toInstruction S320. In Instruction S360, the computer stores the volvedvirtual model for subsequent access by the user. In Instruction S370,the computer and/or the user repeat Instructions S320-S370, for example,for each subsequent medical visit or encounter until treatment of theactual patient is completed.

In another embodiment, the instant invention includes a standard thinclient or other standard client workstation that is programmablyconnected via a computer network to an expert system, such as describedabove. Alternatively, or in addition, the thin client or other standardclient workstation is programmably connected via a computer network toan educational or testing system, such as described above.

In another embodiment, the instant invention includes a knowledge basemodule describing, for example, disease or health state evolution by wayof belief networks or causal probabilistic networks, such as Bayesiannetworks. Advantageously, the knowledge base module enables a user oreducator to update a knowledge base with current medical beliefs andpractices. By way of example, earlier it was believed that ulcers werecaused by certain foods and/or stress levels. Recent studies indicatethat at least some ulcers are caused by bacteria, which should, ofcourse, be treated by an appropriate antibiotic. Such a treatment wouldnot have been recommended or accepted by a knowledge base that onlyreflected the earlier understanding of ulcers. As another example,advances in laboratory test or scanning, which become accepted in thegeneral medical community, are advantageously included in the knowledgebase module. For instance, such advances are included in revealstructures or management plan critiques, which are described usingbelief networks or causal probabilistic networks, such as Bayesiannetworks.

In another embodiment of the instant invention, the instant causalprobabilistic expert systems have medical applications. An example of amedical application includes determining optimal antibiotic selectionsfor an actual patient based at least in part on the patients clinicalcharacteristics and one or more parallel causal probabilistic or beliefnetworks describing health states. Another example of a medicalapplication includes determining a specific chemotherapy regimen amongseveral possible regimens for treating a cancer based, at least in parton, the patient's clinical characteristics and one or more parallelbelief or causal probabilistic networks describing health states.

In an another embodiment of the instant invention, the instant causalprobabilistic expert systems have non-medical applications. An exampleof a non-medical application includes determining credit worthiness forloan approval of an applicant in the financial industry based, at leastin part, on the applicant's suitability characteristics and one or moreparallel causal probabilistic or belief networks describing personalfinancial states. The personal financial states include financial statesrelative to balances and payment history, of for example, car loans,home mortgages, credit cards, student loans, business loans, total assetvalue, cash flow from one or more income sources, and total liabilities.Another example of a non-medical application includes determiningoptimal oil drilling sites based, at least in part, on one or moreparallel causal probabilistic or belief networks describing one or morewildcatters' analysis for identifying potential sites for oil drilling.

The many features and advantages of the invention are apparent from thedetailed specification, and thus, it is intended by the appended claimsto cover all such features and advantages of the invention which fallwithin the true spirit and scope of the invention. Further, sincenumerous modifications and variations will readily occur to thoseskilled in the art, it is not desired to limit the invention to theexact construction and operation illustrated and described, andaccordingly, all suitable modifications and equivalents may be resortedto, falling within the scope of the invention.

1. A method for educating or evaluating a user, comprising the steps of:instantiating a virtual patient for display to the user, the virtualpatient including a plurality of health states; receiving from the userat least one of a query for a medical finding concerning theinstantiated virtual patient and a course of action; and at least oneone of: generating, responsive to the received query, a specific medicalfinding at least in part from a first network defining a health statereveal structure corresponding to the instantiated virtual patient;generating, responsive to the received query, an indication of aninappropriate query, based, at least in part, on a second networkdefining a medical practice management plan; and generating, responsiveto the received course of action, an indication of an inappropriatecourse of action, based, at least in part, on the second network.
 2. Themethod according to claim 1, wherein the medical practice managementplan includes healthcare provider approved medical finding queries.
 3. Acomputer system for evaluating or educating a user, compromising: aprocessor; a computer-readable medium storing instructions executable bysaid processor, said instructions including at least one of: generatinga virtual patient based, at least in part, responsive to a descriptionof a plurality of parallel health state networks; generating the virtualpatient based, at least in part, responsive to a description of rates ofprogression within and/or between the plurality of parallel health statenetworks; and generating patient test data concerning the virtualpatient, based, at least in part, using reveal structures to display ofthe patient test data to patient test data.
 4. The computer systemaccording to claim 3, wherein said computer system is communicatablewith a user via a computer network.
 5. A computer system for evaluatingor educating a user, compromising: a processor; a computer-readablemedium storing instructions executable by said processor, saidinstructions including at least one of: generating a virtual patientbased, at least in part, based on a description of rates of progressionwithin and/or between the plurality of parallel health state networks;and generating patient test data concerning the virtual patient, based,at least in part, on reveal structures to display of the patient testdata.
 6. The computer system according to claim 5, wherein said computersystem is communicatable with a user via a computer network.
 7. Acomputer system for evaluating or educating a user, compromising: aprocessor; a computer-readable medium storing instructions executable bysaid processor, said instructions including at least one of: generatinga virtual patient based, at least in part, on at least one of aplurality of parallel health state networks; generating the virtualpatient based, at least in part, on rates of progression within and/orbetween the at least one of the plurality of parallel health statenetworks; and generating patient test data concerning the virtualpatient, based, at least in part, on reveal structures.
 8. A computerreadable medium including instructions being executed by a computer, theinstructions instructing the computer to execute an educational ortesting system for physicians, the instructions including: (a) accessingat least one representation, which describes at least one of a pluralityof parallel health state networks; (b) scripting a knowledge base, atleast in part, from the at least one representation; and (c)instantiating a model patient, at least in part, from the scriptedknowledge base.
 9. The computer readable medium according to claim 8,wherein the plurality of parallel health state networks describe atleast one of a plurality of primary networks defining diseaseevolutions, a plurality of secondary networks defining risk factorsaffecting progression through a primary network of the plurality ofprimary networks, and a plurality of tertiary networks defining at leastone of causal probabilistic medical complications attributed to at leastone stage in the primary network and medical complications attributed tomanagement of the at least one stage.
 10. The computer readable mediumaccording to claim 8, wherein the instructions further comprise: (d)accessing at least one second representation, which describes rates ofprogression within and/or between the plurality of parallel health statenetworks, and describes task factors that affect the rates ofprogression.
 11. The computer readable medium according to claim 10,wherein the instructions further comprise: (e) accessing at least onethird representation, which supports reveal structures to limit displayof patient test data to patient test data specifically requested by theuser.
 12. The computer readable medium according to claim 11, whereinthe instructions further comprise: (f) accessing at least one fourthrepresentation which supports plan critiques of queries of and treatmentprescribed by the user.
 13. The computer readable medium according toclaim 8, wherein the scripting step (b) includes scripting the knowledgebase, at least in part, from the at least one second representation. 14.The computer readable medium according to claim 11, wherein theinstructions further comprise: (g) receiving one of a course of actionand a query for a specific medical finding concerning the model patientfrom the user responsive to the instantiated model patient; and (h)displaying, if the query is received, the specific medical finding tothe user based at least in part on the at least one thirdrepresentation, and repeating the step (g).
 15. The computer readablemedium according to claim 14, wherein the instructions further comprise:(i) evolving the model patient in accordance with the plurality ofparallel health state networks and responsive to the received course ofaction.
 16. The computer readable medium according to claim 14, whereinthe instructions further comprise: (j) repeating the steps (g) through(i) until the user has completed treatment of the model patient.
 17. Thecomputer readable medium according to claim 16, wherein instructionsfurther comprise: (k) generating a combination of treatment and queriesbased on the at least one fourth representation and the instantiatedmodel patient; and (l) evaluating the query and the treatment by theuser in comparison to the generated optimum combination of treatment andqueries.
 18. A system for educating or evaluating a user comprising: amodel patient generator including a knowledge base scripted from atleast one of at least one first network, which describes a plurality ofparallel health state networks, and at least one second network, whichdescribes at least one rate of progression within and/or between saidplurality of parallel health state networks, and which describes atleast one task factor that affects the at least one rate of progression.19. The system according to claim 18, wherein said patient generatorinstantiating, upon user input, a model patient in a whiteboard, atleast in part, from said scripted knowledge base.
 20. The systemaccording to claim 19, wherein said patient generator receives one of acourse of action and a query for a specific medical finding concerningthe model patient from the user responsive to the instantiated modelpatient, the whiteboard displaying, if the query is received, thespecific medical finding to the user based, at least in part, on atleast one third belief network, which describes at least one patienthealth state reveal structure, the whiteboard evolving the model patientin accordance with the plurality of parallel health state networks andresponsive to the received course of action.
 21. A knowledge base modulefor an educational or testing system or an expert system, comprising atleast one of: at least one first network, which describes each parallelhealth state network of a plurality of parallel health state networks;at least one second network, which describes at least one rate ofprogression within and/or between said plurality of parallel healthstate networks, and which describes at least one task factor thataffects the at least one rate of progression; and at least one thirdnetwork, which describes plan critiques including peer-accepted coursesof action for addressing said plurality of parallel health statenetworks.